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PREFACE 


The  Training  Decision  Technology  Analysis  (TDTA)  is  the  result 
of  a  task  order  project  requested  by  the  Training  Systems  Division 
of  the  Air  Force  Human  Resources  Laboratory.  The  goal  of  this 
effort  was  to  examine  the  technologies  developed  in  the  earlier 
Training  Decisions  System  (TDS)  project,  to  exercise  the  TDS 
software  and  determine  the  usefulness  of  previously-developed  system 
documentation,  to  perform  additional  sensitivity  analyses  to 
establish  the  significance  of  various  input  variables,  and  to 
document  the  relationship  of  TDS  technology  with  the  current 
state-of-the-art  in  various  scientific  disciplines.  The  work  was 
conducted  under  Task  Number  21  of  Contract  Number  F41689-88-D-0251 
with  Metrica,  Incorporated,  San  Antonio,  TX,  as  the  prime 
contractor  and  McDonnell  Douglas  Missile  Systems  Company,  St.  Louis, 
MO,  and  the  CONSAD  Research  Corporation,  Pittsburgh,  PA,  as 
subcontractors . 

This  volume  is  the  final  report  of  this  TDTA  project  and 
provides  a  summary  of  the  work  performed  to  exercise  the  TDS, 
improve  its  software  and  documentation,  and  review  its  relationship 
to  several  scientific  disciplines.  Other  major  products  of  this 
project  include: 

The  TDS  User  Manual.  1  November  1989  (CDRL  A007) . 

The  TDS  Programmer  Guide.  1  November  1989  (CDRL  AO 07) . 

TDS  Examples.  Volumes  1  &  2,  1  November  1989  (to  supplement 

both  manuals) . 


SUMMARY 


The  Training  Decisions  System  (TDS)  is  a  computer-based  decision  support  technology 
which  has  been  developed  to  provide  a  more  integrated  approach  to  Air  Force  training 
planning  and  development.  This  technology  was  developed  in  an  earlier  research  and 
development  (R&D)  effort  which  resulted  in  a  proof-of-concept  system.  The  present  project 
examines  the  technologies  and  scientific  innovations  involved  in  TDS  both  from  the 
standpoint  of  separate  subsystem  technologies  and  from  the  viewpoint  of  an  integrated 
decision  support  system.  The  procedures  and  software  from  the  proof-of-concept  TDS  were 
exercised  to  identify  problems  in  programs  and  software  documentation;  the  problems 
recognized  in  this  exercise  were  corrected  and  new  programs  and  data  files  created. 
Extensive  sensitivity  analyses  were  conducted  to  establish  which  types  of  variables  had  the 
greatest  impact  on  output  products  (total  AFS  training  costs  and  estimates  of  On-the-Job 
Training  (OJT)  capacity  of  representative  field  units.  Additional  validation  work  was  planned 
as  was  a  study  to  examine  the  systematic  biases  in  TDS  data  bases.  The  scientific 
contributions  of  the  TDS  R&D  project  are  examined  and  some  conclusions  drawn  as  to  how 
TDS  relates  to  state-of-the-art  technologies  in  several  academic  and  technical  areas.  The 
potential  interrelationships  of  the  TDS  technology  with  other  Air  Force  R&D  projects  are 
analyzed  and  several  potentially  valuable  interfaces  were  proposed.  Finally,  several  potential 
applications  of  the  TDS  technology  are  recommended  in  terms  of  future  training 
management  research  as  well  as  direct  application  in  other  military  services  and  the  civilian 
sector. 
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TRAINING  DECISIONS  TECHNOLOGY  ANALYSIS 


1.0  INTRODUCTION 

For  nearly  two  decades,  the  Air  Force  has  been  using  the  Instructional  System 
Development  (ISD)  model  to  guide  the  design  of  technical  training  for  enlisted  occupations 
and  for  support  of  new  weapon  system  acquisition  (see  AFR  50-8).  The  ISD  model  requires 
a  systems  approach  to  training  development,  an  approach  aimed  at  providing  "optimal 
training"  for  each  specialty  or  weapon  system  (AFM  50-2,  AFP  50-58).  Many  of  the  data 
elements  required  for  the  application  of  ISD,  however,  are  not  readily  available  in  existing 
Air  Force  data  bases.  Further,  not  all  decision  algorithms  suggested  in  the  ISD  model  have 
been  empirically  validated  nor  are  such  algorithms  equally  applicable  to  all  types  of 
specialties.  Decisions  involving  aircraft  maintenance  specialties,  for  example,  may  require 
an  approach  to  training  decision  making  that  addresses  issues  which  have  a  direct  and 
immediate  impact  on  combat  sortie  generation,  whereas  for  a  support  specialty,  the  issues 
may  involve  primarily  the  cost  per  student  or  student  flow  limitations. 

Recent  developments  in  occupational  analysis  and  training  research,  as  well  as  in  Air 
Force  decision  making  processes  (see  AFR  50-8),  have  created  new  opportunities  for 
optimizing  training.  Such  developments  include  the  recent  emergence  of  Utilization  and 
Training  Workshops  (U&TWs)  and  Training  Planning  Teams  (TPTs)  as  the  primary  vehicles 
for  making  and  coordinating  major  training  decisions  (see  Mitchell,  et  al.,  1987).  Such 
innovative  procedural  changes  also  make  obvious  a  need  for  a  technologically-advanced  data 
generation,  analysis,  and  evaluation  capability. 

To  make  good  decisions  about  the  training  needed  for  an  Air  Force  specialty  or 
system,  decision  makers  must  be  able  to  visualize  and  understand  the  jobs  and  training 
programs  of  the  specialty  or  weapon  system  under  consideration  and  its  technical  training  and 
Professional  Military  Education  (PME)  requirements,  as  well  as  the  relative  costs  and  payoffs 
of  various  training  options.  Such  a  "model"  provides  a  concise  summary  of  the  current  status 
of  the  specialty,  creates  a  common  "language"  for  discussion  or  negotiation,  and  forms  the 
baseline  against  which  various  alternative  proposals  can  be  evaluated.  Thus,  a  modeling 
capability  is  a  t  ery  necessary  and  important  element  needed  for  a  TDS. 

1.1  The  TDS  R&D  Project 

To  provide  adequate  support  for  advanced  training  decision  making,  the  Air  Force 
Deputy  Chief  of  Staff  for  Personnel,  Education  and  Training  (HQ  USAF/DPPE)  requested 
that  the  Air  Force  Human  Resources  Laboratory  develop  a  computer-based  Training 
Decisions  System  (TDS)  to  augment  the  Air  Force  ISD  model  Such  a  system  would 
generate  necessary  front-end  training  requirements  data,  validated  decision  algorithms,  and 
procedures  for  improved  interaction  among  training,  personnel  and  functional  managers. 
The  TDS  would  focus  on  supporting  Air  Force  managers  in  making  decisions  as  to  the  what 
where,  and  when  of  the  training  (including  the  On-the-Job  training)  required  for  a  given 
enlisted  specialty  (Ruck,  1982). 
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Over  several  decades,  the  Air  Force  has  evolved  a  task-based  approach  to  determining 
technical  training  content  and  reviewing  personnel  classification  and  utilization  policies 
(Christal,  1974;  Mitchell,  1988;  Morsh,  1964;  also  see  AFR  8-13).  As  part  of  the  occupational 
analysis  (OA)  process,  tasks  are  defined  by  subject-matter  experts  (SMEs)  of  a  specialty  in 
their  r  vn  technical  terminology,  working  with  analysts  of  the  USAF  Occupational 
Measurement  Center,  Randolph  Air  Force  Base,  Texas  (see  AFR  35-2).  Several  kinds  of 
data  on  these  tasks  are  collected  from  job  incumbents  and  supervisors  for  use  in  reviewing 
training  programs  (see  ATCR  52-22).  Large  samples  of  incumbents  are  asked  to  provide 
information  about  which  tasks  they  perform  in  their  present  jobs  and  the  relative  amount  of 
their  job  time  spent  performing  such  tasks.  These  data  are  used  to  examine  the  variety  of 
specialized  jobs  within  a  specialty  (occupation),  to  assess  how  jobs  change  at  advanced  skill 
levels,  and  to  review  official  specialty  descriptions  and  initial  training  programs  (Christal  & 
Weissmuller,  1988;  Mitchell,  Ruck,  &  Driskill,  1988). 

One  of  the  most  important  data  elements  developed  during  the  OA  process  involves 
noncommissioned  officer  (NCO)  ratings  of  tasks  in  terms  of  recommended  training  emphasis 
for  first-term  and  first-job  airmen.  Such  training  emphasis  (TE)  ratings  have  been  validated 
empirically  using  explanatory  regression  models  in  studies  of  18  AFSs  (Stacy,  Thompson,  & 
Thomson,  1977;  Ruck,  Thompson  &  Stacy,  1987;  Ruck,  Thompson,  &  Thomson,  1978).  Two 
important  findings  of  these  research  studies  were  that  supervisors  agreed  substantially  with 
one  another  on  their  recommendations  in  most  (but  not  all)  career  fields,  and  that 
supervisors’  judgments  were  explainable  in  terms  of  key  ISD  factors.  A  third  important 
finding  was  that  supervisors  could  not  agree  as  to  the  appropriate  sites  for  training  technical 
tasks.  TE  ratings  are  used  operationally  to  evaluate  course  content  of  basic  technical  training 
courses  for  first  enlistment  or  Grstjob  personnel;  typically  they  are  not  used  to  evaluate  field 
training  detachment  (FTD)  or  mobile  training  team  (MTT)  courses  or  OJT  programs 
(Mitchell,  Ruck,  &  Driskill,  1988;  Mitchell,  Sturdevant,  Vaughan,  &  Rueter,  1987;  see  also 
ATCR  52-22).  Hence,  although  methods  had  been  developed  and  validated  for  prioritizing 
AFS  job  tasks  in  terms  of  recommended  training  emphasis  for  first  enlistment  personnel,  no 
reliable  data  were  yet  available  for  determining  appropriate  training  setting  and  site. 

By  1980,  the  determination  of  training  setting  was  being  made  at  U&TWs,  where 
trainers  and  training  managers  met  with  representatives  from  operational  commands  to 
negotiate  training  content  and  training  setting  (Mitchell  et  al.,  1987;  see  also  ATCR  52-15). 
These  conferences  grew  out  of  earlier  procedures  developed  to  bring  initial  skills  technical 
training  in  line  with  initial  job  requirements  ("HASTY  GRAD"  projects),  while  at  the  same 
time  planning  for  those  training  requirements  deferred  to  FIT),  MTT  or  OJT  (Ruck  & 
Birdlebough,  1977;  Vaughan,  1978).  Only  minimal  data  were  available  for  determining 
appropriate  training  settings  for  specialty  tasks;  thus,  these  decisions  were,  of  necessity,  based 
almost  entirely  upon  the  conferees’  personal  experience,  or  on  known  constraints  at  the 
resident  training  school.  For  these  reasons,  many  of  the  decisions  made  in  U&TWs  cannot 
be  consistently  replicated.  In  addition,  no  formal  evaluation  or  estimates  were  made  of  the 
impact  of  such  decisions  on  personnel  utilization,  OJT  costs,  or  mission  performance  (Ruck, 
1982). 


The  Training  Decisions  System  (IDS)  project  was  initiated  by  the  Air  Force  Human 
Resources  Laboratory  in  1983,  at  the  request  of  Headquarters,  United  States  Air  Force  and 
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with  the  support  of  Air  Training  Command,  to  develop  a  computer-based,  training  decision 
support  system  to  aid  Air  Force  managers  in  making  critical  training  decisions.  The  purpose 
of  this  research  and  development  (R&D)  project  was  to  explore  the  feasibility  of  such  a 
system,  develop  the  technologies  required  to  gather,  estimate,  or  process  data  which  could 
help  Air  Force  managers  make  more  realistic  decisions  concerning  training  requirements  of 
a  specialty,  and  to  evaluate  the  possible  consequences  of  such  decisions. 

The  proof-of-concept  Training  Decisions  System  (TDS)  developed  in  the  initial  R&D 
is  a  computer-based  decision  support  system  which  was  developed  by  the  Human  Factors 
Branch,  Systems  Engineering  and  Analysis  Department  of  McDonnell  Douglas  Astronautics 
Company  (now  the  McDonnell  Douglas  Missile  Systems  Company),  St.  Louis,  MO,  and  the 
CONSAD  Research  Corporation,  Pittsburgh,  PA.  (with  some  early  participation  by  the 
Research  Triangle  Institute  of  Research  Triangle  Park,  North  Carolina,  and  some  final 
testing  support  by  the  MAXIMA  Corporation,  San  Antonio,  TX).  This  research  and 
development  was  conducted  under  AFHRL  Contract  Number  F336 15-83-0028  from 
September  1983  through  October  1988.  The  details  of  how  the  system  was  developed  and 
how  it  operates  are  available  in  a  separate  technical  report  (see  Vaughan,  Mitchell,  Yadrick, 
Perrin,  Knight,  Eschenbrenner,  Rueter,  &  Feldsott,  1989). 


12  TDS  Technology 

The  emphasis  in  the  summary  report  of  the  initial  R&D  project  was  on  how  the 
system  was  developed  and  its  final  architecture,  processes,  and  operations.  Inherent  in  that 
report  were  a  number  of  new  and  sometimes  innovative  technologies  which  had  been 
developed  to  meet  the  needs  of  a  new  approach  to  Air  Force  training  decisions.  However, 
some  of  these  technologies  were  not  very  visible  in  the  final  report  due  to  the  need  to 
summarize  five  years  of  development  as  well  as  to  communicate  what  the  TDS  is  and  what 
it  does. 

The  details  of  how  these  new  technologies  were  developed  and  the  potential  scientific 
contributions  of  the  proof-of-concept  R&D  were  somewhat  scattered  throughout  a  series  of 
formal  technical  reports  and  papers,  draft  CDRL  reports,  technical  presentations,  and 
published  proceedings  of  various  meetings  and  conferences. 

Given  this  situation,  there  was  clearly  a  need  to  bring  together  into  one  document,  the 
evidence  of  the  technologies  involved  in  the  TDS  and  to  assess  the  potential  utility  of  such 
technologies  for  the  purposes  they  were  designed.  Their  possible  uses  in  other  Air  Force 
R&D  projects  need  to  be  examined.  There  was  also  a  need  to  explore  other  technologies 
for  potential  applications  in  the  TDS  as  well  as  any  possible  refinements  of  the  new 
technologies  from  the  proof-of-concept  system. 

The  present  volume  has  been  developed  to  help  meet  such  needs.  It  will  review  the 
TDS  in  terms  of  technologies  used  and  how  they  operate.  It  summarizes  a  number  of  tests 
of  the  system  and  identifies  a  number  of  refinements  which  need  to  be  made  to  make  the 
system  more  efficient  or  to  enhance  its  potential  applications.  It  examines  how  sensitive  the 
TDS  is  to  variations  in  a  number  of  types  of  input  data  and  design  constraints. 
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This  report  also  examines  the  scientific  contributions  of  the  TDS  R&D  in  terms  of  the 
state-of-the-art  of  several  of  the  academic  and  technical  areas  involved  (cost  accounting, 
operations  research  modeling,  learning  theory,  job  and  occupational  analysis,  etc.).  This 
report  will  also  examine  some  of  the  potential  interfaces  of  the  TDS  with  other  AFHRL 
R&D  projects  both  from  the  standpoint  of  using  TDS  data  in  those  systems  or  using  new 
R&D  findings  in  the  TDS.  Finally,  this  report  examines  potential  applications  of  the  TDS 
for  use  by  other  military  services,  federal  agencies,  and  civilian  organizations. 

To  accomplish  all  these  goals,  it  is  first  necessary  to  have  some  perspective  on  the 
various  types  of  technologies  involved  in  the  proof-of-concept  TDS.  The  following  chapter 
examines  the  TDS  in  terms  of  each  of  the  TDS  subsystems  as  well  as  an  integrated 
technology. 


2.0  A  DECISION  SUPPORT  SYSTEM  FOR  AIR  FORCE  TRAINING  DECISIONS 


For  over  two  decades,  the  Air  Force  has  been  using  the  Instructional  System 
Development  (ISD)  model  to  guide  the  design  of  technical  training  for  enlisted  occupations 
and  for  support  of  new  weapon  system  acquisition  (see  AFR  50-8).  The  ISD  model  requires 
a  systems  approach  to  training  development,  an  approach  aimed  at  providing  "optimal 
training"  for  each  specialty  or  weapon  system  (AFM  50-2,  AFP  50-58).  Many  of  the  data 
elements  required  for  the  application  of  ISD,  however,  are  not  readily  available  in  existing 
Air  Force  data  bases.  Further,  not  all  decision  algorithms  suggested  in  the  ISD  model  have 
been  empirically  validated  nor  are  such  algorithms  equally  applicable  to  all  types  of 
specialties.  Decisions  involving  aircraft  maintenance  specialties,  for  example,  may  require 
an  approach  to  training  decision  making  that  addresses  issues  which  have  a  direct  and 
immediate  impact  on  combat  sortie  generation,  whereas  for  a  support  specialty,  the  issues 
may  involve  primarily  the  cost  per  student  or  student  flow  limitations. 

Recent  developments  in  occupational  analysis  and  training  research,  as  well  as  in  Air 
Force  decision  making  processes  (see  AFR  50-8  and  AFR  50-23)  have  created  new 
opportunities  for  optimizing  specialty  training.  Such  developments  include  the  recent 
emergence  of  Utilization  and  Training  Workshops  (U&TWs)  and  Training  Planning  Teams 
(TPTs)  as  the  primary  vehicles  for  making  and  coordinating  major  training  decisions 
(Mitchell,  et  al.,  1987).  Such  innovative  procedural  changes  also  make  obvious  a  need  for 
a  technologically-advanced  data  generation,  analysis,  and  evaluation  capability.  To  make 
good  decisions  about  the  training  needed  for  an  Air  Force  specialty  or  system,  decision 
makers  must  be  able  to  visualize  and  understand  the  jobs  and  training  programs  of  the 
specialty  or  weapon  system  under  consideration  and  its  technical  training  and  Professional 
Military  Education  (PME)  requirements,  as  well  as  the  relative  costs  and  payoffs  of  various 
training  options.  Such  a  "model"  provides  a  concise  summary  of  the  current  status  of  the 
specialty,  creates  a  common  "language"  for  discussion  or  negotiation,  and  forms  the  baseline 
against  which  various  alternative  proposals  can  be  evaluated.  Thus,  a  modeling  capability 
is  a  very  necessary  and  important  element  needed  for  good  decision  making.  Some  type  of 
decision  support  system  is  needed  to  provide  information  to  Air  Force  decision  makers  to 
help  them  assess  potential  outcomes  of  various  alternatives. 

2.1  Decision  Support  Systems 

Recent  technological  developments  in  personal  computers,  computer  networks,  and 
computer-based  models  have  made  possible  a  level  of  interactive  support  for  decision  makers 
never  before  possible.  A  body  of  literature  and  theory  has  developed  in  recent  years  in  this 
new  area,  generated  in  part  by  those  interested  in  electronic  data  processing  and  computer 
science  and  in  some  measure  from  those  concerned  with  management  science  and 
information  systems  (Sprague  and  Carlson,  1982:4)  or  artificial  intelligence  and  information 
economics  (Keen  and  Scott  Morton,  1978:38-44).  Another  major  influence  has  been 
behaviorists  concerned  with  cognitive  research  (Newell  and  Simon,  1972;  Fleishman  and 
Quaintance,  1984),  managerial  effectiveness  (Campbell,  Dunnette,  Lawler,  and  Weick,  1970) 
or  decision  strategies  for  dealing  with  various  levels  of  uncertainty  (MacCrimmon  and  Taylor, 
1976).  All  of  these  areas  have  been  integrated  into  modern  decision  support  technology. 
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2.1.1  Key  Concepts  of  Decision  Support  Systems 


Decision  support  for  management  decision  making  implies  the  use  of  computers  to: 

1.  Assist  managers  in  their  decision  processes  in  semi-structured  tasks. 

2.  Support,  rather  than  replace,  managerial  judgment. 

3.  Improve  the  effectiveness  of  decision  making  rather  than  its  efficiency. 

(Keen  and  Scott  Morton,  1978:1) 

Systems  designed  to  provide  support  for  decision  making  are  generally  called  Decision 
Support  Systems  (DSS)  which  are  very  different  from  the  traditional  record  keeping  and 
transaction  processing  uses  of  computers  (Sprague  and  Carlson,  1982:xiii).  A  DSS  can  be 
defined  as  follows: 

"A  DSS  is  a  coherent  system  of  computer-based  technology  (hardware, 
software,  and  supporting  documentation)  used  by  managers  as  an  aid  to  their 
decision  making  in  semistructured  decision  tasks.  We  stress  supporting  rather 
than  replacing  managerial  judgments  (Bennett,  1983:1)." 

The  key  concepts  involved  in  computer-based  support  for  executive  decision  making  were 
first  developed  by  M.  S.  Scott  Morton  in  the  early  1970’s  who,  at  that  time,  termed  them 
"management  decision  systems"  (Scott  Morton,  1971).  The  DSS  label  has  become  more 
widely  used  during  the  later  1970’s  and  early  1980’s  (Keen  and  Scott  Morton,  1978:1).  DSSs 
are  said  to  be  typically  "interactive  computer-based  systems  which  help  decision  makers  utilize 
data  and  models  to  solve  unstructured  problems.  The  unique  contribution  of  DSS  resulted 
from  the  key  italicized  words  (Sprague  and  Carlson,  1982:4)."  These  authors  note  that  few 
actual  systems  completely  satisfy  this  rather  restrictive  definition.  They  go  on  to  say  that  a 
DSS  may  be  defined  by  its  capabilities  in  several  critical  areas,  including  the  following: 

"They  tend  to  be  aimed  at  the  less  well  structured,  underspecified  problems  that 
upper-level  managers  typically  face. 

They  attempt  to  combine  the  use  of  models  or  analytic  techniques  with 
traditional  data  access  and  retrieval  functions. 

They  specifically  focus  on  features  that  make  them  easy  to  use  by  noncomputer 
people  in  an  interactive  mode. 

They  emphasize  flexibility  and  adaptability  to  accommodate  changes  in  the 
environment  and  decision-making  approach  of  the  user." 

(Sprague  and  Carlson,  1982:6) 
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In  examining  the  use  of  computers  to  support  various  types  of  decision  making,  Bennett 
(following  the  ideas  of  Anthony  about  planning  and  control  systems)  classifies  decisions  into 
the  following  four  categories: 

Strategic  planning:  decisions  relating  to  setting  policies,  choosing  objectives  and 
selecting  resources. 

Management  control:  decisions  related  to  assuring  effectiveness  in  acquisition 
and  use  of  resources. 

Operational  control:  decisions  related  to  assuring  effectiveness  in  performing 
operations 

Operational  performance:  decisions  that  are  made  while  performing  the 
operations. 

(Carlson,  1983:16) 

Gorry  and  Scott  Morton  have  been  concerned  with  the  degree  to  which  decisions  were 
structured  or  unstructured;  they  combined  Anthony’s  categories  with  the  dimension  of 
structure  to  develop  a  four  by  four  matrix  of  decision  categories  (Gorry  and  Scott  Morton, 
1971).  In  describing  their  approach,  Carlson  notes  that,  "almost  no  computer  support  is  used 
for  unstructured  decisions.  They  argue  that  the  semi-structured  and  unstructured  decisions 
(especially  management  control  and  strategic  planning)  are  of  the  greatest  concern  to 
decision  makers  (Carlson,  1983:16)".  Decision  Support  Systems  are  designed  to  assist 
managers  with  such  semi-structured  and  unstructured  types  of  decisions,  while  more 
structured  decisions  (such  as  budget  analysis)  are  serviced  by  management  information 
systems  (MIS).  Some  authors  consider  DSSs  as  a  subset  of  the  broader  MIS  classification. 

2.1.2  Critical  Characteristics  of  DSSs 

DSSs  have  high  potential  value  for  managers  and  other  decision  makers,  yet  there  have 
been  only  a  few  actually  successful  applications  of  computer-assisted  decision  support  systems 
in  government  and  business,  although  there  are  a  good  number  of  effective  management 
information  systems.  Based  on  a  review  of  recent  citations  in  the  PsycLIT  Abstracts  (APA), 
some  of  the  more  successful  applications  of  DSS  have  been  in  academic  administration, 
library  science,  R&D  management,  program  evaluation,  small  group  assisted  decision  making, 
and  strategic  planning.  Many  more  are  proposed  than  are  successfully  implemented.  DSSs 
sometimes  fall  into  disuse  or  are  misused  either  because  of  manager’s  concerns  over  loss  of 
responsibility.  The  reliability  of  data  (because  of  defensive  reporting,  cognitive  biases,  or  fear 
of  responsibility)  is  sometimes  a  problem  which  can  result  in  abandonment  of  the  DSS  unless 
suitable  reliability  control  methods  are  developed  (Zakay,  1982). 

Alter  (1980)  reports  a  survey  of  fifty-six  DSSs  which  seemed  to  fall  into  two  broad 
categories:  data-oriented  systems  and  model-oriented  systems.  Data-oriented  systems  are 
characterized  by  data  retrieval,  analysis,  and  presentation  functions,  whereas  model-oriented 
systems  use  simulation  or  optimization  models  to  help  managers  and  executives  make  more 
effective  decisions.  Both  of  these  types  of  functions  or  capabilities  are  needed  if  decision 
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makers  are  to  make  use  of  the  full  potential  of  the  integrated  DSS  approach.  These 
functions  were  developed  to  parallel  the  processes  observed  in  manual  decision  making. 

In  reviewing  studies  on  decision  making  in  order  to  specify  realistic  DSS  requirements, 
Carlson  (1983:20)  concluded  that  there  are  a  variety  of  decision  making  processes  so  that  a 
DSS  must  be  able  to  support  multiple  processes.  He  also  notes  that  different  types  of 
decisions  have  different  data  and  processing  requirements  so  that  a  DSS  must  be  flexible  in 
order  to  support  various  types  of  decisions. 

Keen  and  Scott  Morton  observe  that  a  DSS  approach  has  potential  impact  as  follows: 

a.  The  impact  is  on  decisions  in  which  there  is  sufficient  structure  for 
computer  and  analytic  aids  to  be  of  value  but  where  managers’  judgement  is 
essential. 

b.  The  payoff  is  in  extending  the  range  and  capability  of  managers’  decision 
processes  to  help  them  improve  their  effectiveness. 

c.  The  relevance  for  managers  is  the  creation  of  a  supportive  tool,  under  their 
own  control,  which  does  not  attempt  to  automate  the  decision  process, 
predefine  objectives,  or  impose  solutions. 

(Keen  and  Scott  Morton,  1978:2) 

Thus,  to  be  an  effective  tool,  a  DSS  must  be  flexible  in  terms  of  the  types  of  decisions 
which  it  can  support  and  must  be  carefully  designed  so  that  direction  of  the  system  is  undeT 
the  control  of  the  user.  The  user  must  specify  the  questions  to  be  addressed  and  the  DSS 
must  be  able  to  provide  the  data  in  whatever  form  that  can  be  used  most  effectively  by  the 
user.  In  this  regard,  Carlson  (1983:20)  reviewed  the  literature  to  identify  the  attributes  or 
characteristic  needs  of  decision  makers  and  made  the  following  observations: 

1.  Decision  makers  rely  on  conceptualizations  in  making  a  decision,  and  a 
DSS  should  provide  familiar  representations  (e.g.,  charts  and  graphs)  to  assist 
in  conceptualization. 

2.  Decision  makers  perform  Intelligence,  Design,  and  Choice  activities  while 
making  a  decision,  so  a  DSS  should  provide  operations  which  support  these 
activities. 

3.  Decision  makers  need  memory  aids,  so  a  DSS  should  provide  memory  aids 
which  help  carry  out  the  decision-making  process. 

4.  Decision  makers  exhibit  a  variety  of  skills,  styles,  and  knowledge,  so  a  DSS 
should  help  decision  makers  work  in  their  own  idiosyncratic  ways. 

5.  Decision  makers  expect  to  control  their  decision  support,  so  a  DSS  should 
provide  control  aids  which  help  decision  makers  exercise  direct,  personal 
control. 
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From  these  observations,  Carlson  concluded  that  representations  are  a  key  focus  of  an 
effective  DSS.  The  system  needs  to  assist  managers  in  developing  their  conceptualization  of 
the  problems  so  as  to  reflect  as  close  as  possible  the  real  world  issues  and  constraints.  This 
might  be  done  through  charts  or  graphs  or  whatever  other  type  of  display  a  manager  is 
comfortable  using  so  long  as  it  incorporates  the  significant  variables  on  which  realistic 
decision  are  to  be  made.  For  group  decision  making,  such  "modeling"  of  reality  also  needs 
to  extend  the  decision  makers’  understanding  of  the  problem  (multiple  perspectives,  multiple 
criteria)  by  serving  as  a  memory  aid  for  complex  data  bases  (available  for  recall  as  needed). 

Odiome  (1984)  observes  that  key  decision  makers  also  need  to  be  better  motivated  by 
the  organization;  they  need  to  be  treated  in  some  respects  as  "stars."  A  DSS  can  do  this  in 
a  nonintrusive  way,  by  demonstrating  a  considerable  investment  on  the  part  of  the 
organization  in  collecting  information,  building  data  bases,  and  providing  a  complex  decision 
support  system  to  help  or  assist  its  key  decision  makers. 

Thierauf  (1982),  in  discussing  the  ten  essential  characteristics  of  a  DSS,  observes  that  the 
system  must  take  a  broad-based  approach  to  supporting  decision  making,  with  an  emphasis 
on  managing  by  perception.  By  this  he  means  that  a  DSS  must  take  a  broader  view  of  the 
organization  and  must  be  forward  looking;  decision  makers’  judgments  are  absolutely  critical 
for  understanding  future  external  and  internal  trends.  The  complex  threads  of  such  trends 
must  be  assimilated  in  the  context  of  competing  or  alternative  probable  decisions  (on  long 
term  organizational  goals,  focus  of  marketing,  etc.).  He  also  suggests  that  a  DSS  must  make 
use  of  appropriate  mathematical  and  statistical  models  in  a  flexible  way.  By  making  use  of 
on-line  computing  capacity  and  flexible  models,  managers  can  insert  their  own  assumptions 
about  key  characteristics  or  issues;  multiple  runs  are  made  to  assess  outcomes  under  a  variety 
of  conditions  or  assumptions.  He  states,  "Thus,  instead  of  a  single  optimum  result  from  the 
model  (as  in  MIS),  several  outputs  are  obtained  to  answer  *what  if  questions.  This  enables 
solutions  to  be  judged  by  managers  on  a  variety  of  criteria,  including  risk,  robustness,  and 
performance  (Thierauf,  1982:67)." 

Other  requirements  noted  by  Thierauf  include:  output  directed  to  organizational 
personnel  at  all  levels;  integrated  subsystems;  comprehensive  database;  easy-to-use  approach; 
and  an  adaptive  system  over  time.  These  are  "essential  characteristics"  of  an  effective  DSS 
(Thierauf,  1982:61-76)  which  permit  it  to  meet  the  needs  of  organizational  decision  makers. 
Since  his  essential  characteristics  are  critical  to  the  design  of  an  effective  DSS,  they  are 
recapped  in  Figure  2.1  on  the  next  page. 

Sprague  and  Carlson  (1982)  also  call  for  flexibility  in  both  building  at  DSS  and  in  its 
employment.  They  urge  an  interactive  design  and  creation  of  an  adaptive  system  which  can 
be  modified  in  use  to  better  meet  the  needs  of  its  users  or  new  sets  of  decision  makers 
(Sprague  and  Carlson,  1982:139-144).  They  also  suggest  user  tutorials  to  enhance  the  ability 
of  decision  makers  to  employ  the  system  as  well  as  an  evaluation  and  tracking  system  which 
can  provide  needed  information  to  improve  the  system.  Their  view  is  that  these  processes 
and  mechanisms  can  provide  a  flexibility  for  a  DSS  which  is  missing  from  most  traditional 
management  information  systems  and  thus  greatly  enhances  the  probability  that  the  DSS  can 
and  will  be  used  for  effective  decision  making.  They  assert  that  such  flexibility  must 
encompass  both  the  level  of  technology  and  the  time  horizon  for  change  (Ibid:  145). 
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1.  Broad-based  approach  to  supporting  decision  making  -  accent  on  ’‘manage¬ 
ment  by  perception.”  Decision  support  systems  go  beyond  capabilities  of  a  typical 
management  information  system  by  taking  a  broad  view  of  the  organization  in  terms  of 
supporting  decisions.  They  utilize  “management  by  perception,”  whereby  managers  are 
assisted  in  perceiving  important  future  trends  and  helped  in  adapting  the  organization  to 
upcoming  conditions. 

X  Human/machine  interface  where  human  retains  control  over  the  decision- 
making  process.  The  utilization  of  CRT  terminals  gives  the  decision  maker  the  capa¬ 
bility  to  retrieve,  manipulate,  present,  and  store  data  such  that  there  is  a  human/machine 
dialogue  during  decision  making.  Throughout  the  interface,  the  decision  maker  has  com¬ 
plete  control  over  all  stages  of  the  decision-making  process  in  solving  a  problem. 

3.  Support  decision  making  for  solving  structured,  semistructured,  and  unstruc¬ 
tured  problems.  Generally,  the  focus  is  on  semistructured  and  unstructured  decisions 
although  well-structured  decisions  can  also  be  made  in  a  DSS  environment.  Basically, 
DSS  recognizes  the  need  for  bringing  together  human  judgment  and  computerized  infor¬ 
mation  for  improving  the  quality  of  the  final  decision. 

4.  Utilization  of  appropriate  mathematical  and  statistical  models.  Based  on 
the  needs  of  the  problem  being  solved,  one  or  more  mathematical  and/or  statistical 
models  are  employed  to  assist  the  decision  maker  in  evaluating  alternative  solutions.  The 
real  payoff  from  mathematical  and  statistical  models  as  well  as  modeling  languages 
comes  from  integrating  them  into  the  decision  support  system  as  decision  tools. 

3.  Query  capabilities  to  obtain  information  by  request  -  interactive  mode.  In 
DSS.  query  capabilities  go  beyond  those  of  interactive  computation  and  include  those  of 
responsiveness.  The  latter  item  refers  to  utilizing  the  system  as  an  extension  of  the  in¬ 
dividual's  reasoning  process  throughout  the  decision-making  process. 

6.  Output  directed  to  organization  personnel  at  all  levels.  Although  DSS  has 
the  capability  to  supply  top  and  middle  management  with  important  short-  to  long-range 
planning  information  for  decision  making  (that  was  not  available  with  earlier  computer 
systems),  it  is  also  capable  of  providing  lower  management  and  their  operating  personnel 
with  the  necessary  output  for  supporting  decision  on  controlling  current  operations. 

7.  Integrated  subsystems.  This  concept  refers  to  the  capability  or  processing 
data  for  use  by  all  subsystems  along  broader,  functional  lines  rather  than  the  traditional, 
narrow  departmental  lines.  Integrated  subsystems  allow  managers  and  their  personnel  to 
retrieve  and  manipulate  information  of  concern  to  them  for  supporting  decisions. 

8.  Comprehensive  data  base.  Contents  of  the  data  base  for  DSS  must  go  be¬ 
yond  just  providing  historical  information  about  current  and  past  operations.  It  must 
also  contain  appropriate  external  information  that  is  compatible  with  internal  informa¬ 
tion  contained  in  the  data  base.  Generally,  it  is  desirable  to  utilize  a  data  base  manage¬ 
ment  system  (DBMS)  to  assist  in  a  human/machine  dialogue. 

9.  Easy-to-use  approach.  The  hallmark  of  effective  DSS  is  that  it  is  easy  to 
use,  that  is.  not  only  does  it  assist  the  decision  maker  in  supporting  decisions  via  a 
human/machine  interface,  but  also  allows  the  individual  to  pursue  his  or  her  own  natural 
tendencies  to  problem  solving.  From  this  view,  the  individual  feels  comfortable  and  “at 
home”  with  the  system  rather  than  intimidated  by  it. 

10.  Adaptive  system  over  time.  The  main  thrust  of  the  adaptive  system  con¬ 
cept  is  that  the  decision  maker  is  able  to  confront  changing  conditions  and  adapt  the 
system  to  meet  these  changes.  The  time  factor  for  effecting  system  changes  is  a  few 
weeks  to  several  months. 


Figure  2.1  Essential  Characteristics  of  Decision  Support  Systems. 
(Adapted  from  Thierauf,  1982:Figure  3-8,78) 
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2.1.3  Potential  DSS  Design  Issues 


Moore  and  Chang  have  noted  that  there  are  potential  problems  in  designing  an  effective 
DSS,  "given  the  organizational  and  technological  constraints  inherent  in  any  computer-based 
development  effort  (Moore  and  Chang,  1983:173)."  These  include  the  issue  of  system  or 
problem  migration,  where  both  system  design  and  problem  understanding  shift  over  time. 
Another  potential  issue  is  subset  evolution;  this  involves  expanding  of  system  capabilities  over 
time  to  match  the  decision  maker’s  changing  preferences  and  increasing  capacity  to  use  the 
system.  There  is  also  some  conflict  between  "soft"  versus  "hard"  DSS  capabilities  where 
initial,  generalized  capabilities  are  increasingly  specified  by  designers  or  users  so  as  to 
become  very  specific  ("hard")  system  functions.  A  final  issue  involves  a  weak-strong  design 
continuum;  a  weak  design  follows  users’  current  preferences  where  a  strong  design 
deliberately  attempts  to  shape  and  refine  the  users’  derisinn-malring  process  (Ibid.:  174). 
Each  of  these  issues  requires  some  consideration  on  the  part  of  DSS  designers  and  specific 
choices  must  be  made  in  terms  of  practicality  for  a  given  organization  and  with  due  regard 
for  cost  (development  time,  equipment,  etc.).  Further,  controversy  over  these  and  other 
operational  concepts  often  make  it  difficult  for  systems  designers  to  construct  a  simple 
practical  system  since  typically  too  many  optional  capabilities  are  thought  to  be  desirable. 

These  authors  maintain  that  the  most  popular  concepts  involving  DSS  (such  as  improving 
"managerial  effectiveness")  are  not  very  helpful  for  a  DSS  designer.  They  have  developed 
an  alternative  framework  for  DSS  design  including  a  new  definition  of  a  DSS: 

Decision  Support  System  (DSS)  -  an  extensible  system  with  intrinsic  capability 
to  support  ad  hoc  data  extraction,  analysis,  consolidation  and  reduction,  as 
well  as  decision-modeling  activities.  For  example,  a  general  ledger-based 
planning  system  with  both  preformatted  and  user-definable  reports  or 
forecasts  (loosely  interpreted  as  models)  is  a  DSS. 

(Moore  &  Chang,  1983:179) 

Inherent  in  this  definition  is  the  recognition  of  the  need  for  a  DSS  to  evolve  over  time 
as  users  learn  how  to  make  use  of  the  system  and  system  designers  learn  more  about  how 
decision  makers  can  use  models,  data  summaries,  and  alternative  analyses  to  improve  their 
decision  making  processes.  In  this  respect,  it  is  expected  that  there  will  be  a  synergistic  effect 
where  successful  implementation  of  a  DSS  will  generate  a  body  of  information  about 
advanced  decision  making  which  will  lead  to  further  R&D  not  just  to  improve  the  DSS  but 
also  which  will  lead  to  new  decision  support  technologies. 

Zachary  takes  this  approach  a  step  further  and  maintains  that  a  DSS  must  be  designed 
so  as  to  extend  the  cognitive  skills  of  its  user.  He  believes  that,  in  general,  decision  support 
technology  has  developed  more  rapidly  than  its  theory;  he  views  a  DSS  as  an  interface 
between  the  cognitive  processes  of  its  user  and  a  set  of  underlying  computation  algorithms. 
For  his  perspective,  a  DSS  is  defined  as:  "any  interactive  system  that  is  specifically  designed 
to  improve  the  decision  making  of  its  user  by  extending  the  user’s  cognitive  decision-making 
abilities  (Zachary,  1988:998)." 
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He  goes  on  to  note  that, 

"it  must  do  more  than  simply  place  information  within  the  reach  of  the 
decision  maker,  hoping  that  such  access  will  somehow  assist  decision  making. 

...  A  DSS  is  intended  to  remove  some  obstacle  or  relax  some  constraint  that 
is  preventing  the  human  decision  maker  from  making  the  best  possible 
decision." 

(Zachary,  1988:998)." 

In  this  context,  Zachary  notes  that  most  decision  support  systems  are  situation  specific 
and  "there  are  few  (if  any)  generic  systems.  Virtually  all  are  tied  to  a  specific  decision 
problem,  decision  environment  and/or  class  of  decision  maker  and  specific  class  of 
difficulties/constraints  affecting  it/them  (Ibid.)." 

Recognizing  these  facts,  he  contends  that  DSS  design  must  focus  on  two  major 
components  -  the  human-computer  interface  (HCI)  and  the  decision-aiding  algorithms.  The 
decision-aiding  algorithms  produce  all  the  information  that  is  displayed  and  the  user  never 
sees  the  data  structures,  inference  algorithms,  optimization  programs  or  other  procedures. 
"As  far  as  the  DSS  user  is  concerned,  the  interface  is  the  system  (Zachary,  1988:999).  Both 
areas  are  important  parts  of  a  DSS,  and  this  author  notes  a  number  of  specific  design  issues 
which  must  be  considered  in  designing  such  a  system  (such  as  selecting  automated  analysis 
and  reasoning  techniques,  representation  aids,  judgement  refinement  and  amplification  tools, 
etc).  He  concludes  by  calling  for  a  "Human-Aided  Optimization"  to  deal  with  complex,  non¬ 
linear  functions,  so  as  to  maximize  the  strengths  of  both  computer  systems  and  experienced 
human  decision  makers  (Ibid.: 1028). 

2.1.4  Future  R&D  Requirements 

While  one  primary  focus  of  any  future  training  decision  technology  R&D  will  obviously 
be  directed  toward  improving  the  system-user  interface,  a  major  possibility  is  the  evolution 
of  the  system  toward  an  Artificial  Intelligence  (AI)  technology.  Gony  and  Krumland  (1983) 
have  anticipated  this  line  of  development  and  suggest  that  AI  research  has  already  had  an 
impact  on  DSS  technologies  and  designs.  In  discussing  the  contribution  of  artificial 
intelligence  research  to  DSS,  they  note  that: 

The  recognition  of  the  importance  of  knowledge  in  problem  solving  has 
sharpened  the  study  of  the  intellectual  topography  of  problem  domains.  The 
study  of  the  knowledge  involved  in  tasks  such  as  analyzing  a  scene  or 
understanding  a  sentence  has  in  turn  stimulated  the  development  of 
representational  schemes  and  new  programming  languages  for  the 
organization  and  marshaling  of  different  kinds  of  knowledge  in  problem¬ 
solving  programs.  In  this  way,  a  range  of  problems  are  being  attacked  with 
renewed  vigor  in  the  field. 

(Gorry  and  Krumland,  1983:216-217) 

These  authors  conclude  that  a  theory  of  knowledge  will  eventually  be  of  concern  to  DSS 
designers  as  it  is  to  AI  researchers,  and  that  the  continued  development  ad  hoc  decision 
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systems  will  eventually  accumulate  information  necessary  for  the  development  of  a 
comprehensive  theory  of  knowledge.  They  do  believe  that  AI  technology  can  and  should  be 
applied  in  advanced  DSS  applications. 

In  reviewing  recent  DSS  development  approaches,  Stabell  (1983)  cautions  that  the  focus 
of  new  DDS  approaches  must  focus  on  decisions  (the  D  in  DSS).  He  writes. 

It  is  the  decision  in  the  concept  that  should  define  the  unique  context  of  DSS, 
and  the  decision  should  have  implications  for  the  why,  how,  and  what  of 
building  such  systems.  However,  in  much  of  the  literature  I  find  that  the 
decision  is  not  really  in  focus.  In  many  discussions  very  little  would  be  altered 
if  we  were  to  exchange  the  decision  label  for  another  term  (management, 
executive,  personal,  mind,  or  whatever). 

(Stabell,  1983:223) 

He  recommends  a  dual  focus  in  future  R&D  efforts  not  only  to  develop  enhanced  DSS 
technologies  but  also  to  study  decisions  themselves  and  how  the  decision  process  works  and 
changes  across  time.  His  approach  to  developing  a  DSS  is  summarized  in  Figure  22.  He 
recommends  a  "decision-oriented  approach"  where  a  second  major  focus  is  on  research  on 
the  decisions  themselves  and  suggests  that  this  impacts  on  DSS  design  and  development 
processes.  Only  by  such  a  dual  thrust  can  substantial  long-term  progress  be  made  in 
improving  not  only  decisions  being  made  but  the  decision-making  process  itself. 

Another  major  area  of  possible  R&D  is  that  of  Utility  Analysis.  Most  research  of  utility 
models  has  concentrated  on  the  efficiency  of  personnel  selection  algorithms  (c.f.,  Blum  & 
Naylor,  1968;  Boudreau  &  Rynes,  1985).  In  recent  years,  however,  utility  analysis  has  also 
been  applied  in  the  evaluation  of  training  programs,  with  an  emphasis  on  the  gain  in 
productivity  of  trained  versus  untrained  workers  (c £,  Schmidt,  F.L,  Hunter,  JJEL,  & 
Pearlman,  1982).  There  is  some  question,  however,  as  to  the  impact  Minch  such  utility 
analysis  efforts  have  had.  Zedeck  and  Cascio  observe  that 

An  unfortunate  myth  that  has  developed  among  many  business  decision 
makers  is  that  employee  selection  methods  and  other  behavioral  science-based 
organizational  interventions  have  had  little  impact  on  workforce  productivity, 
in  part  because  statistical  indices  of  utility  have  had  little  impact  on  decision 
makers.  Like  it  or  not,  the  language  of  business  is  dollars,  not  forecasting 
efficiency,  r2,  or  improved  performance  in  terms  of  standard  deviation. 

(Zedeck  &  Cascio,  1984:490491) 

In  their  review  of  personnel  decisions,  these  authors  note  that  utility  analysis  has 
considerable  promise,  particularly  since  it  is  now  possible  to  translate  the  outcomes  of 
personnel  decisions  (including  training)  into  dollars,  the  common  metric  for  business.  They 
caution,  however,  that  the  issues  are  generally  more  complex  than  present  utility  models 
might  suggest  They  dte  Boudreau’s  (1983)  concern  that  current  utility  models  do  not 
recognize  the  economic  effects  of  variable  costs,  corporate  taxes,  and  disamnt  rates. 
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Figure  22  Decision-Oriented  DSS  Development  Process  (From  Stabell,  1983:234). 

Some  researchers  have  extended  the  Schmidt,  et  al.  approach  to  take  into  account  the 
factors  of  concern  to  Boudreau.  Mathieu  and  Leonard,  for  example,  report  development  of 
a  "time-based"  approach  which  corrects  utility  estimates  to  account  for  such  economic 
considerations  (Mathieu  &  Leonard,  1987).  Their  methodology  permits  examination  of  the 
total  dollar  savings  resulting  from  training  for  periods  ranging  from  one  to  twenty  years  in 
the  future.  They  also  built  into  their  model  the  capability  to  vary  potentially  significant 
variables  and  assumptions;  for  example,  they  were  able  to  assess  alternative  assumptions  - 
one  where  the  effectiveness  of  training  remains  constant  over  time  and  an  alternative  where 
there  is  a  25  percent  decay  in  training  effectiveness  per  year  (Ibid.:327,  Table  4). 

Others,  such  as  Becker  (1989),  have  become  increasingly  concerned  that  while  utility 
analysis  can  now  "more  accurately  measure  the  costs  and  benefits  of  human  resources 
programs,  both  in  the  present  and  the  future" ...  "conventional  utility  models  have  ignored  the 
effects  of  the  external  labor  market  on  estimated  utilities  (Becker,  1989:531)."  He  concludes 
that  failing  to  consider  such  factors  generally  overstates  the  likely  utility  of  the  programs 
being  evaluated. 

If  such  labor  market  factors  are  also  to  be  accounted  for  in  a  utility  model  in  addition 
to  the  economic  factors  of  Boudreau  and  time  variance  (Mathieu  &  Leonard),  such  utility 
models  quickly  become  extremely  complex  and  the  results  difficult  to  analyze  and  understand. 
Many  of  these  factors  or  assumptions  may  be  interactive.  As  Zedeck  &  Casdo  observed: 
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So  far,  theoretical  developments  in  this  area  have  outpaced  practical 
applications.  One  theoretical  question  that  should  receive  greater  empirical 
attention  in  different  organizational  contexts  is  the  relationships  among  the 
utilities  of  selection,  training,  and  other  organizational  interventions.  As 
Schmidt,  et  al.  (1982b)  and  Landy,  et  al.  (1982a)  have  noted,  they  may  be 
interactive  or  additive,  depending  on  such  parameters  as  the  selection  ratio, 
selection  validity,  and  SDe.  To  be  sure,  these  questions  need  to  be  addressed 
in  the  context  of  an  overall  personal  decision  system. 


(Zedeck  &  Casdo,  1984:493) 

In  a  more  recent  review  of  training-related  psychological  research,  Latham  (1988)  takes 
a  fairly  critical  approach  to  utility  analysis.  He  states: 

Little  has  appeared  on  the  utility  or  cost  effectiveness  of  training  programs 
per  se.  It  may  be  noteworthy  that  the  economists  themselves  have  not 
embraced  this  methodology  as  advocated  by  psychologists.  What  is 
troublesome  about  work  in  this  area  is  that  the  standard  deviation  of 
performance  in  dollars  is  almost  always  based  on  a  primitive  estimate  (Landy 
et  al.,  1982).  Dreher  &  Sackett  (1983)  have  taken  a  skeptical  view  by  pointing 
out  that  (a)  there  is  no  evidence  that  a  rational  estimate  approach  to  assessing 
the  standard  deviation  of  performance  approximates  the  true  value;  (b) 
agreement  among  job  experts  is  not  a  guarantee  that  the  estimates  are  valid; 
and  (c)  the  procedure  lacks  face  validity  in  that  the  basis  of  each  supervisor’s 
judgment  is  unknown.  As  is  the  case  with  performance  appraisal  (Latham, 

1986),  the  question  arises  of  who  in  the  organization  is  requesting  information 
on  the  monetary  value  of  conducting  in-house  training  programs?  More 
appropriate  may  be  to  ask  "What  does  upper  management  take  into  account 
when  determining  the  value  of ...  training?" 

An  implicit  assumption  is  that,  having  conducted  a  utility  analysis,  a  company 
will  eliminate  fewer  personnel  programs  (e.g.  training)  during  an  economic 
hardship  than  it  would  have  before.  This  assumption  needs  to  be  tested.  The 
reason  why  Brogden  &  Taylor’s  (1950)  work  on  the  dollar  criterion  has  been 
largely  ignored  for  38  years  may  be  that  the  customer  has  not  asked  for  it 

(Latham,  1988:560-561) 

This  alleged  lack  of  interest  on  the  part  of  managers  and  decision  makers  may  stem  from 
the  lack  of  context  (i.e.,  framework  of  total  training  costs)  or  the  absence  of  an  adequate 
decision  support  system  (tools  to  assist  in  decision  making)  rather  than  any  disregard  for 
potential  dollar  savings.  As  noted  in  the  introduction,  some  $30  billion  a  year  is  spent  on 
formal  training  and  approximately  $180  billion  on  on-the-job  training  (ASTD,  n.&).  Much 
of  the  informal  OJT  costs  are  hidden  costs,  which  are  not  normally  visible  to  managers.  Yet 
formal  training  programs  have  their  greatest  impact  by  avoiding  such  OJT  costs  by  providing 
more  most-effective,  centralized  training.  Any  improvement  in  the  management  of  OJT  has 
the  potential  for  significant  savings  and  substantially  enhanced  productivity. 
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In  order  for  a  manager  or  decision  maker  to  come  to  realistic  decisions,  any  DSS  must 
first  provide  the  decision  maker  with  a  good  perspective  on  current  training  programs  and 
their  costs,  including  the  normally  invisible  OJT  requirements  and  costs.  Once  such  a 
baseline  is  established  (context,  if  you  will),  the  decision  maker  can  then  more  properly  assess 
the  real  magnitude  or  impact  of  any  new  or  proposed  training  program. 

At  present,  utility  analysis  is  probably  not  yet  at  a  stage  where  most  of  the  important 
variables  and  assumptions  have  been  adequately  researched  or  their  interactions  understood. 
The  many  functions  and  their  interactions  are  extremely  complex  and  are  probably  beyond 
the  capabilities  of  most  decision  makers,  without  some  decision  support.  As  further  R&D 
is  accomplished  on  utility  analysis  (UA),  on  the  human  decision  making  process,  or  on 
decision  support  technologies,  the  research  findings  may  be  mutually  beneficial.  Indeed,  as 
realistic  DSSs  are  developed  and  their  real  world  applications  effected,  then  we  may  find  that 
the  essential  elements  of  UA  can  be  extended  to  include  a  much  more  realistic  set  of 
problems  and  data. 

Future  TDS  R&D  efforts  may  well  represent  such  an  opportunity  for  the  further  UA 
R&D  since  it  involves  large  number  of  people  and  training  programs,  high  levels  of  cost,  and 
real  public  policy  issues  which  could  benefit  from  improved  decision  making. 

2.1.5  The  I  DS  as  a  DSS 

Clearly,  the  problems  faced  by  most  Air  Force  training  decision  makers  are  much  the 
same  as  those  encountered  by  many  civilian  executives  in  business  and  public  service.  The 
kinds  of  information  needed  before  realistic  decisions  can  be  made  and  the  varying  levels  of 
specificity  involved  are  typical  of  the  types  of  issues  faced  by  designers  of  most  decision 
support  systems.  Many  of  the  modeling  techniques  and  conceptualization  procedures 
developed  in  recent  years  through  DSS  R&D  efforts  should  be  directly  applicable  in  facing 
the  complex  issues  involved  with  Air  Force  training  programming  and  planning.  Most  of  the 
lessons  learned  by  DSS  developers  are  relevant  for  Air  Force  decision  making  as  well. 

For  this  review  of  the  technology  involved  in  the  TDS,  one  of  the  issues  to  be  examined 
is  how  the  TDS  R&D  effort  relates  to  the  typical  development  process  for  a  DSS.  What 
does  the  DSS  literature  and  recent  theoretical  developments  imply  for  the  implementation 
and  future  R&D  for  TDS?  What  are  the  accomplishments  of  the  TDS  R&D  in  terms  of 
implications  for  the  DSS  area? 

Before  we  can  begin  to  answer  these  questions,  we  need  to  first  briefly  review  the  TDS 
project  to  develop  an  overview  of  what  the  project  was  designed  to  do  and  how  it 
accomplished  its  stated  requirements.  In  this  review,  the  emphasis  will  be  on  explicating  the 
various  technologies  used  to  meet  the  needs  of  this  R&D  project,  and  how  these  technologies 
relate  to  the  corresponding  academic  disciplines.  In  addition,  we  must  examine  how  this 
project  relates  to  other  on-going  Air  Force  R&D  efforts. 
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22  The  TDS  R&D  Project 


To  provide  adequate  support  for  advanced  training  decision  making,  the  Air  Force 
Deputy  Chief  of  Staff  for  Personnel,  Education  and  Training  (HQ  USAF/DPPE)  requested 
that  the  Air  Force  Human  Resources  Laboratory  develop  a  computer-based  Training 
Decisions  System  (TDS)  to  augment  the  Air  Force  ISD  model.  Such  a  system  would 
generate  necessary  front-end  training  requirements  data,  validated  decision  algorithms,  and 
procedures  for  improved  interaction  among  training,  personnel,  and  functional  managers. 
The  TDS  would  focus  on  supporting  Air  Force  managers  in  making  decisions  as  to  the  what 
where,  and  when  of  the  training  (including  the  On-the-Job  training)  required  for  a  given 
enlisted  specialty  (Ruck,  1982). 

Over  several  decades,  the  Air  Force  has  evolved  a  task-based  approach  to  determining 
technical  training  content  and  reviewing  personnel  classification  and  utilization  policies 
(Christal,  1974;  Mitchell,  1988;  Morsh,  1964;  also  see  AFR  8-13).  As  part  of  the  occupational 
analysis  (OA)  process,  tasks  are  defined  by  subject-matter  experts  (SMEs)  of  a  specialty  in 
their  own  technical  terminology,  working  with  analysts  of  the  USAF  Occupational 
Measurement  Center,  Randolph  Air  Force  Base,  Texas  (see  AFR  35-2).  Several  kinds  of 
data  on  these  tasks  are  collected  from  job  incumbents  and  supervisors  for  use  in  reviewing 
training  programs  (see  ATCR  52-22).  Large  samples  of  incumbents  are  asked  to  provide 
information  about  which  tasks  they  perform  in  their  present  jobs  and  the  relative  amount  of 
their  job  time  spent  performing  such  tasks.  These  data  are  used  to  examine  the  variety  of 
specialized  jobs  within  a  specialty  (occupation),  to  assess  how  jobs  change  at  advanced  skill 
levels,  and  to  review  official  specialty  descriptions  and  initial  training  programs  (Christal  & 
Weissmuller,  1988;  Mitchell,  Ruck,  &  Driskill,  1988). 

One  of  the  most  important  data  elements  developed  during  the  OA  process  involves 
noncommissioned  officer  (NCO)  ratings  of  tasks  in  terms  of  recommended  training  emphasis 
for  first-term  and  first-job  airmen.  Such  training  emphasis  (TE)  ratings  have  been  validated 
empirically  using  explanatory  regression  models  in  studies  of  18  AFSs  (Stacy,  Thompson,  & 
Thomson,  1977;  Ruck,  Thompson  &  Stacy,  1987;  Ruck,  Thompson,  &  Thomson,  1978).  Two 
important  findings  of  these  research  studies  were  that  supervisors  agreed  substantially  with 
one  another  on  their  recommendations  in  most  (but  not  all)  career  fields,  and  that 
supervisors’  judgments  were  explainable  in  terms  of  key  ISD  factors.  A  third  important 
finding  was  that  supervisors  could  not  agree  as  to  the  appropriate  sites  for  training  technical 
tasks.  TE  ratings  are  used  operationally  to  evaluate  course  content  of  basic  technical  training 
courses  for  first  enlistment  or  first  job  personnel;  typically  they  are  not  used  to  evaluate  field 
training  detachment  (FTD)  or  mobile  training  team  (M  i  l  )  courses  or  OJT  programs 
(Mitchell,  Ruck,  &  Driskill,  1988;  Mitchell,  Sturdevant,  Vaughan,  &  Rueter,  1987;  see  also 
ATCR  52-22).  Hence,  although  methods  had  been  developed  and  validated  for  prioritizing 
AFS  job  tasks  in  terms  of  recommended  training  emphasis  for  first  enlistment  personnel,  no 
reliable  data  were  yet  available  for  determining  appropriate  training  settings  and  sites. 

By  1980,  the  determination  of  training  settings  was  being  made  at  utilization  and  training 
workshops  (U&TWs),  where  trainers  and  training  managers  met  with  representatives  from 
operational  commands  to  negotiate  training  content  and  training  settings  (Mitchell  et  al., 
1987;  see  also  ATCR  52-15).  These  conferences  grew  out  of  earlier  procedures  developed 
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to  bring  initial  skills  technical  training  in  line  with  initial  job  requirements  ("HASTY  GRAD" 
projects),  while  at  the  same  time  planning  for  those  training  requirements  deferred  to  FID, 
MTT  or  OJT  (Ruck  &  Birdlebough,  1977;  Vaughan,  1978).  Only  minimal  data  were 
available  for  determining  appropriate  training  settings  for  specialty  tasks;  thus,  these  decisions 
were,  of  necessity,  based  almost  entirely  upon  the  conferees’  personal  experience,  or  on 
known  constraints  at  the  resident  training  school.  For  these  reasons,  many  of  the  decisions 
made  in  U&TWs  cannot  be  consistently  replicated.  In  addition,  no  formal  evaluation  or 
estimates  were  made  of  the  impact  of  such  decisions  on  personnel  utilization,  OJT  costs,  or 
mission  performance  (Ruck,  1982). 

2.2.1  Approach 

The  general  strategy  used  in  developing  the  Training  Decisions  System  focused  on  first 
defining  functional  requirements  and  then  developing  the  structure,  methodologies,  and 
procedures  for  meeting  those  requirements.  The  research  team  collected  as  much 
information  as  possible  concerning  Air  Force  traini;  g  decisions  and  how  they  were  made;  a 
variety  of  agencies  and  offices  were  visited  to  define  the  desired  functions  and  data 
requirements  of  the  system  (Mitchell,  Sturdevant,  Vaughan,  &  Rueter,  1987). 

Based  on  the  information  gathered  from  such  visits  and  a  review  of  the  previous  research 
literature,  the  following  types  of  variables  were  determined  to  be  particularly  important  for 
Air  Force  decision  makers  and  thus  must  be  represented  explicitly  within  the  TDS: 

o  Tasks  of  the  Specialty  and  Their  Associated  Characteristics 
o  Task  Allocations  to  Training  Settings 

o  Managers’  Preferences  for  Task  Allocations  to  Training  Settings 
o  Times  Required  to  Training  Tasks  in  Various  Setting  Allocations 
o  Utilization  and  Training  Patterns,  in  terms  of: 

-  Jobs  &  Associated  Tasks 

-  Training  States 

-  Transition  Probabilities  Among  Jobs  and  Training  States 

-  Numbers  of  Airmen  in  Various  Training  and  Job  States 

-  Airman  Proficiency  States 
o  Training  Costs 

o  Training  Resource  Requirements 
o  Training  Capacities 
o  Managers’  Preferences  for  U&T  Patterns 

Some  data  on  the  tasks,  jobs,  and  training  states  are  generally  available  from  existing 
sources,  such  as  the  occupational  analysis  program  or  course  documentation.  For  other 
variables,  few  if  any  data  are  available  in  existing  Air  Force  data  bases.  For  example,  data 
concerning  OJT  costs  are  not  routinely  available,  and  innovative  approaches  were  required 
to  develop  such  information  for  use  by  training  decision  makers. 

A  preliminary  integrated  systems  design  was  developed  to  guide  the  R&D  effort  which 
provided  for  a  set  of  subsystems  to  deal  with  the  three  broad,  relatively  independent  classes 
of  data  (task  information,  training  and  job  patterns,  and  costs  versus  training  capacity 
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constraints)  plus  a  fourth  subsystem  to  integrate  such  data  and  display  it  for  decision  makers. 
The  preliminary  system  design  was  validated  and  refined  through  interactions  with  potential 
Air  Force  users  (Vaughan,  Yadrick,  Perrin,  Cooley,  Dunteman,  Clark,  &  Rueter,  1984). 

The  general  approach  used  in  TDS  development  was  to  start  from  known  data  bases, 
such  as  the  OA  report  files  or  personnel  flow  statistics,  and  to  develop  new  data  gathering 
technologies  to  determine,  estimate,  or  approximate  other  required  information.  Where 
possible,  an  evaluation  of  alternative  approaches  was  conducted;  several  methods  were 
initially  tested  and  the  method  yielding  the  best  results  was  adopted  for  use  in  the  TDS. 
Where  necessary,  experimental  designs  were  employed  when  it  was  necessary  to  verify 
method  effects  or  analyze  data  differences.  Generally,  data  collection  and  analysis  methods 
were  developed  on  two  Air  Force  specialties  (Avionic  Inertial  and  Radar  Navigation  Systems, 
AFS  328X4,  and  Security  and  Law  Enforcement,  AFS  811XX),  and  were  later  validated  and 
refined  on  two  additional  specialties  (Electronic  Computer  and  Switching  Systems,  AFS 
305X4,  and  Aircraft  Environmental  Systems,  AFS  423X1). 

Such  development  and  testing  was  done  in  a  systematic,  integrated  way  to  insure  that  the 
various  types  of  information  could  be  synthesized  into  a  coherent  picture  of  the  various  ways 
training  requirements  of  a  specialty  could  be  met  and  the  relative  cost  of  each  alternative. 
The  objective  was  to  make  visible  to  specialty  managers  and  training  staff  officers  their 
decision  options,  as  well  as  the  constraints  and  relative  cost  consequences  of  each  decision. 

At  periodic  intervals  throughout  the  TDS  development  project,  the  research  and 
development  team  briefed  their  progress  and  results  to  an  Air  Force  TDS  advisory  panel, 
which  included  representatives  of  HQ  USAF,  HQ  Air  Training  Command,  the  USAF 
Occupational  Measurement  Center,  functional  specialty  managers,  and  training  managers. 
Such  periodic  progress  reviews  were  extremely  constructive  in  terms  of  constructive  critique 
of  results  and  positive  interaction  between  researchers  and  potential  system  users. 
Suggestions  were  made  for  needed  design  improvements,  data  displays,  new  data  sources  to 
be  evaluated,  or  other  possible  solutions  to  research  and  development  problems.  As  a  result 
of  such  interactions,  a  separate  plan  was  developed  to  guide  the  transition  of  TDS  from  an 
research  and  development  project  to  an  operational  system  in  an  appropriate  Air  Force 
organization;  this  transition  plan  was  circulated  to  various  Air  Force  agencies  and  offices  for 
coordination  and  staffing  (Vaughan,  Yadrick,  Perrin,  Mitchell,  Sturdevant,  Rueter,  &  Ward, 
1985).  The  end  result  of  such  interactive  progress  reviews  was  an  improved  systems  design 
and  TDS  products  which  should  be  much  more  useful  to  potential  TDS  users. 

2.2.2  Subsystem  Technologies  in  the  Proof-of-Concept  TDS 

The  initial  research  and  development  of  the  TDS  has  been  completed.  The 
proof-of-concept  system  consists  of  three  interdependent  subsystems  which  deal  with  Task 
Characteristics,  Field  Utilization  pattern  modeling,  and  Cost/Resource  data  generation,  as 
well  as  a  fourth  Integration  and  Optimization  subsystem  (see  Figure  23).  This  section  of  the 
report  will  present  a  conceptual  overview  of  the  system  to  provide  a  general  perspective  of 
the  TDS  and  a  basic  understanding  of  its  structure  and  operation.  For  the  purposes  of  the 
present  report,  a  short  description  of  each  subsystem  and  it  development  should  suffice. 
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Figure  23.  Major  Subsystems  of  the  Training  Decisions  System 


22.2.1.  Task  Characteristics  Subsystem  (TCS)  Technologies 

The  TCS  embodies  two  separate  technologies  dealing  with  tasks  and  groups  of  tasks;  one 
involves  grouping  of  tasks  and  the  second  allocates  these  groups  of  tasks  to  training  settings. 

The  Task  Module  (TM)  is  a  grouping  of  tasks  which  can  serve  as  the  basic  unit  of 
analysis  in  the  TDS.  TMs  solve  a  number  of  problems  associated  with  the  use  of  task-level 
data  (see  Perrin,  et  al.,  1987).  As  noted  earlier,  the  Air  Force  presently  makes  some  use  of 
task-based  data  for  training  decisions;  one  problem  is  that  many  tasks  share  a  common  skill 
and  knowledge  base  and  thus  much  task-level  information  may  be  redundant  for  training 
decision  making.  TM-level  data,  reflecting  shared  skills  and  knowledges,  reduces  the 
possibility  of  redundancy  and  overestimating  training  requirements.  A  second  part  of  the 
problem  in  using  task  data  is  the  fact  that  each  specialty  involves  300  to  2,000+  tasks,  far  too 
many  for  managers  to  process  in  a  typical  U&TW  session.  Indeed,  U&TW  participants 
generally  focus  on  review  of  the  relevant  Specialty  Training  Standard  (STS)  topics  or  areas, 
leaving  detailed  review  of  tasks  to  occupational  analysts  and  training  developers. 

According  to  some  interpretations  of  the  ISD  literature,  in  order  to  fully  understand  and 
classify  all  the  tasks  of  a  specialty  in  terms  of  their  common  skills  and  knowledges,  a  detailed 
task  analysis  would  be  required  on  every  task  before  any  decisions  about  training  could  be 
made.  The  fact  is  that  task  analysis  is  an  exceedingly  time-consuming,  labor-intensive, 
expensive  process,  and  various  types  of  tasks  may  require  differing  types  of  analysis  (DeVries, 
Eschenbrenner,  &  Ruck,  1980;  Eschenbrenner,  DeVries,  Miller,  &  Rude,  1980).  The  Air 
Force  probably  cannot  afford  the  manpower  and  expense  of  a  detailed  task  analysis  for  every 
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task  of  every  specialty.  Therefore,  a  procedure  is  needed  to  group  or  cluster  tasks  which 
share  common  skills  and  knowledges;  i.e.,  those  tasks  which  could  be  trained  together  most 
efficiently. 

Task  Clustering  Techniques.  In  TDS,  individual  tasks  are  grouped  into  clusters  of  related 
tasks  called  Task  Modules  (Perrin,  Vaughan,  Yadrick,  Mitchell,  &  Knight,  1986).  Advanced 
Comprehensive  Occupational  Data  Analysis  Programs  (CODAP)  procedures  were  used  for 
hierarchical  clustering  of  tasks  and  interpretation  of  meaningful  clusters  of  tasks  on  the  basis 
of  task  co-performance.  It  was  found  that  the  help  of  subject  matter  experts  (SMEs)  was 
critical  to  naming  the  modules,  assessing  their  significance  in  terms  of  common  skills  and 
knowledges  (i.e.,  groupings  of  tasks  which  can  be  effectively  trained  together),  and  allocating 
isolated  tasks  which  did  not  statistically  cluster.  The  final  procedure  was  a  combined 
approach  using  statistical  co-performance  clustering  to  form  initial  groupings  of  tasks,  having 
an  analyst  identify  task  groupings  which  appeared  meaningful,  and  then  having  SMEs  title 
and  refine  the  task  groupings  into  TMs.  This  approach  saves  time  for  both  analysts  and 
SMEs  and  provides  a  structured  focus  for  their  efforts  (see  Perrin  et  al.,  1986  for  details). 

Allocation  Functions.  A  second  TCS  technology  involves  the  development  of  Allocation 
Curves  as  a  mechanism  for  translating  between  hours  of  training  provided  and  the  degree  of 
required  proficiency  achieved  for  each  Task  Module  in  each  type  of  training  setting. 

Once  TMs  are  finalized,  survey  instruments  are  developed  to  gather  information  as  to 
how  TMs  are  or  should  be  allocated  to  various  training  settings.  Groups  of  senior 
technicians  in  the  specialty,  who  are  familiar  with  the  work  of  the  specialty,  estimate  how 
much  training  time  is  currently  devoted  to  reach  minimum  required  proficiency  for  the 
various  groups  of  tasks  for  the  following  training  settings:  classroom,  correspondence  courses 
such  as  career  development  courses  (CDCs),  hands-on  training  (FTDs,  MTTs,  etc.), 
supervised  hands-on  experience  on  the  job  (OJT),  and  other  programs.  These  raters  also 
provide  training  time  estimates  for  "ideal"  training  (i.e.,  the  most  effective  mix  of  types  of 
training).  Finally,  the  raters  are  asked  how  long  it  would  take  in  each  setting  to  train  the 
TM,  if  the  training  was  provided  in  only  that  setting  (e.g.,  only  classroom  training,  etc.).  This 
"maximum  effective  training"  may  not  always  yield  full  proficiency;  it  may  not  be  possible  to 
fully  train  a  TM  in  one  or  more  of  the  settings  alone.  In  such  cases,  respondents  indicated 
the  proficiency  level  reached  as  a  percent  of  full  proficiency.  Thus,  each  SME  provided  six 
allocation  judgments  for  each  TM:  four  that  related  the  maximum  training  time  in  each  of 
four  settings  to  the  proficiency  level  reached,  one  for  the  current  allocation,  and  one  for  the 
most  preferred  allocation  of  training. 

What  Is  Proficiency?  For  TDS,  proficiency  was  defined  as  a  percentage  of  the  training 
needed  by  an  average  individual  to  reach  the  minimum  required  standard  for  each  TM  (the 
"go/no  go"  level  of  OJT  =  100%  proficiency).  SMEs  generally  understood  and  were  able 
to  estimate  degrees  of  proficiency  expressed  in  this  way.  SMEs  are  also  able  to  reliably  and 
consistently  describe  the  current  training  pattern  in  terms  of  how  training  in  each  type  of 
setting  contributes  incrementally  to  the  attainment  of  full  proficiency  (i.e.,  the  partial 
proficiency  achieved).  They  also  easily  conceptualized  other  combinations  of  training  for 
reaching  the  same  goal  (i.e.,  alternative  training  allocations).  Because  these  combinations 
vary  systematically,  they  can  be  expressed  as  mathematical  functions. 
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It  was  hypothesized  that  proficiency  gain  from  training  in  a  setting  would  be  greatest 
initially  and  would  decline  as  more  training  was  provided  in  that  setting.  Eventually,  there 
would  be  no  more  gain  from  providing  training.  Thus,  the  predicted  relationship  between 
proficiency  and  time  in  a  training  setting  is  that  of  initial  gain  followed  by  proficiency 
leveling-off;  a  negatively  accelerated  curve.  This  general  set  of  relationships  is  depicted  in 
Figure  2.4;  these  curves  are  theoretical  estimates  to  illustrate  the  expected  relationships.  The 
relationships  can  also  be  expressed  in  the  following  polynomial  regression  equation: 


Proficiency  =  a  *  class-hours  -  b  *  class-hours2  +  c  *  self-study-hours  -  d  * 
self-study-hours2  +  e  *  field-training-hours  -  f  *  field-training-hours2 
+  g  *  work-hours  -  h  *  work-hours2, 

where  "a"  through  "h"  are  coefficients  to  be  estimated  by  multiple  regression,  2  indicates 
squaring,  and  the  regression  equation  is  constrained  to  pass  through  the  origin  (there  is  no 
constant  for  the  Y  intercept). 

This  model  involves  specific  hypotheses  about  the  nature  of  the  relationship  between 
setting  training  hours  and  proficiency.  Specifically,  controlling  for  training  in  each  of  the 
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Figure  2.4.  Hypothesized  Relationship  Between  Honrs  of  Training  in  a  Setting 
and  Proficiency  Gain  (from  Perrin  et  al.,  1988:29). 


22 


other  training  settings,  the  first-order  parameter  is  specified  to  be  positive  and  the 
second-order  parameter  is  negative,  yielding  the  predicted  negatively  accelerated  curve. 
Across  the  four  AFSs  studied  during  TDS  development,  this  statistical  model  was  strongly 
supported.  Statistical  estimates  consistent  with  the  polynomial  regression  equation  were 
found  in  well  over  90%  of  the  allocation  curves  in  all  four  specialties  (Perrin  et  al.,  1988). 

There  are  two  additional  sources  of  support  for  the  conceptualization  of  proficiency  gain 
as  a  negatively  accelerated  function  of  training  time  in  a  setting.  First,  the  overall  fit  of  the 
polynomial  regression  model  was  found  to  be  quite  good,  averaging  over  65%  (multiple  R2) 
in  two  specialties.  Second,  the  additional  variance  explained  by  the  second-  order  terms  in 
the  allocation  equations  was  substantial  (approximately  15%  increase  in  R2  for  these  two 
specialties),  indicating  that  simple  linear  functions  are  not  sufficient  to  describe  proficiency 
gains  from  training  in  each  setting  (see  Figure  2.5  for  an  example  allocation  curve).  As 
noted  in  the  report  of  our  research  and  development  of  the  TCS,  a  curvilinear  model  is  much 
more  descriptive  of  these  relationships  (Perrin  et  al.,  1988). 

Possible  alternative  allocations  range  (theoretically)  from  training  everything  about  a  TM 
in  the  classroom  to  training  the  TM  entirely  on  the  job.  (In  reality,  it  is  likely  that  complete 
proficiency  can  only  be  achieved  through  a  combination  of  training  programs;  only  in  OJT 
could  100%  of  the  required  proficiency  be  achieved  in  a  single  setting.)  By  collecting  data 
from  SMEs  on  a  few  possibilities  (i.e.,  current,  "ideal,"  and  "maximum  effective"  training)  and 
on  the  time  involved  for  each,  allocation  curves  are  generated  which  represent  all  possible 
combinations.  Such  curves  permit  a  direct  translation  among  settings  of  training  time  to 
percent  of  required  proficiency  for  each  TM,  thus  facilitating  specification  and  evaluation  of 
training  alternatives  (i.e.,  various  combinations  of  training  times  in  different  settings  that,  in 
total,  achieve  100%  of  the  required  proficiency). 

Allocation  curves  for  all  TMs  of  a  specialty  derived  through  this  survey  approach  give  the 
TDS  maximum  flexibility  in  considering  different  ways  of  dividing  training  among  training 
settings,  as  well  as  identifying  the  limits  of  each  type  of  training.  In  addition,  by  asking  SMEs 
to  provide  information  on  "ideal"  and  "maximum  effective"  training  times  in  the  various 
settings,  we  are  able  to  evaluate  the  limits  of  effective  training  for  any  given  setting, 
independent  of  any  training  in  the  other  settings.  [As  additional  training  settings  are 
identified,  they  could  be  added  to  the  data  collection  effort,  and  to  the  AFS  data  base;  the 
only  limitation  is  on  the  number  of  settings  SMEs  can  rate  realistically.] 

This  capability  serves  as  the  foundation  for  developing  and  evaluating  alternative  patterns 
of  training  (as  will  be  discussed  in  the  Field  Utilization  Subsystem  section).  Thus,  the 
allocation  curves  are  a  very  significant  part  of  the  overall  TDS  design.  Their  development 
and  validation  represent  a  substantial  advance  in  training  decisions  technology,  although  it 
is  obvious  that  further  empirical  validation  of  these  relationships  is  also  needed. 

The  set  of  TMs  developed  for  a  specialty  represents  one  of  the  major  building  blocks  of 
the  TDS.  The  TMs  are  a  major  input  to  the  Field  Utilization  Subsystem  (FUS)  and  serve 
as  the  basis  for  describing  jobs  and  training  states  (courses,  OJT  programs,  etc.).  Data  are 
summarized  or  averaged  to  the  module  (TM)  level,  such  that  jobs  can  be  described 
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Training  Hours 

Figure  2.5.  Example  Allocation  Curves  for  Aircraft  Environmental 
Systems  (AFS  423X1)  TM  34,  Doppler  Sensor  Control  Boxes. 


in  terms  of  percent  of  job  incumbents  performing  each  TM  and  the  total  time 
spent  on  TMs.  Training  course  content  is  also  expressed  in  terms  of  hours  of  training  per 
TM.  Thus,  training  and  job  content  descriptions  (and  data  files)  share  a  common  set  of 
terms.  The  TMs  also  serve  as  a  foundation  for  the  Resource/Cost  Subsystem  (RCS)  in  that 
information  about  training  resources  required  and  available  are  collected  on  a  TM  basis. 
This  greatly  reduces  the  complexity  of  required  RCS  data  collection  and  the  statistical 
prediction  of  training  costs  and  capacities. 

2.2.2.2  Field  Utilization  Subsystem  (FUS)  Technologies 

The  FUS  provides  information  for  defining  training  and  job  assignment  patterns  and 
collecting  management  preference  values  for  the  current  and  several  plausible  alternative 
approaches  to  training,  assigning,  and  utilizing  airmen  in  a  particular  specialty  over  the  span 
of  their  Air  Force  careers  (see  Vaughan,  et  al.,  1989;  Yadrick,  et  al.,  1987).  This  complex 
area  required  the  development  of  several  new  technologies  in  order  to  meet  the  needs  of  the 
system. 

FUS  Flow  Simulation  Program:  A  Dynamic  Modeling  Approach.  A  computer-based 
simulation  program  was  developed  for  TDS  as  a  tool  to  facilitate  analysis  of  current  and 
alternative  U&T  patterns,  and  to  permit  calculation  of  total  specialty  training  requirements 
for  each  pattern.  Such  a  simulation  program  is  absolutely  necessary  for  the  IDS  if  we  are 
to  model  and  evaluate  changes  to  any  of  the  variables  used  to  characterize  specialty  jobs,  job 
content,  training  programs,  course  content,  assignment  probabilities,  training  capacities  of 
units,  and  training  costs  or  resources. 
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The  dynamic  simulation  program  is  a  major  innovation  in  technology  in  terms  of  better 
estimation  of  OJT  requirements  for  all  jobs  in  the  specialty.  It  considerably  extends  earlier 
research  aimed  at  adapting  econometric  and  manpower  modeling  to  the  issues  of  personnel 
assignment  flows  and  training  costs  in  operational  units  (Eisele,  Bell,  &  Laidlaw,  1978; 
Rueter,  Bell,  &  Malloy,  1980).  Such  a  simulation  is  critical  for  calculating  OJT  requirements, 
costs,  and  realistic  representative  unit  training  capacities;  any  changes  in  job  content,  training 
programs,  or  even  assignment  probabilities  have  an  impact  either  directly  or  indirectly  on  the 
requirement  for  OJT. 

The  program  is  a  dynamic  simulation  system  which  processes  descriptive  data  files  in  such 
a  way  as  to  show  total  numbers  of  personnel  flowing  through  jobs  and  training  programs  over 
an  extended  period  of  time.  Data  can  be  examined  by  any  specified  period  of  time;  training 
flows  are  generally  expressed  as  annual  rates  where  assignment  probabilities  are  more 
realistically  portrayed  for  two  or  three  year  intervals  equivalent  to  the  typical  Air  Force 
specialty  job  assignment.  Such  dynamic  processing  of  several  models  (U&T  patterns)  of  a 
specialty  provides  the  needed  flexibility  to  project  future  requirements  and  assess  the 
consequences  of  proposed  changes. 

Existing  computer-based  simulation  programs,  such  as  the  Simulation  Language  for 
Alternative  Modeling  (SLAM),  were  found  to  be  not  fully  adequate  for  handling  the  complex 
specialty  models  to  the  degree  needed  in  the  TDS.  Several  systems  were  examined  and 
tested  but  were  found  to  be  unable  to  handle  the  complexities  of  most  Air  Force  specialties 
(Yadrick  et  al.,  1987).  Thus,  new  simulation  software  had  to  be  designed  and  tested  which 
would  meet  the  specific  requirements  of  the  TDS. 

Modeling  an  Air  Force  Specialty 

In  the  TDS,  once  an  analyst  has  defined  a  set  of  task  modules  (TMs)  for  an  occupation 
and  collected  data  on  training  setting  allocations,  these  data  are  then  used  to  describe  all 
training  courses  and  jobs  in  a  relatively  concise  way.  A  static  picture  of  the  jobs  and  training 
programs  is  not  enough,  however,  to  capture  the  dynamics  of  the  normal  flow  of 
reassignments  of  people  from  one  job  to  another,  nor  to  assess  the  probability  of  receiving 
advanced  technical  training  or  professional  military  education  (Mitchell  et  al.,  1988). 
Considerable  training  is  made  necessary  by  such  reassignments;  the  amount  varies 
dramatically  depending  on  the  prior  training  and  experience  of  the  individual  and  the  type 
of  equipment  (system  or  mission)  in  place  at  new  locations.  The  TDS  integrates  assignment 
probability  data  with  job  and  training  data  into  a  dynamic  model  for  the  occupation,  and 
employs  a  simulation  program  to  estimate  total  hours  of  training  required  for  each  TM  for 
each  person  in  all  courses  and  on-the-job  training  (OJT)  programs.  This  paper  provides  an 
overview  of  the  job  and  training  pattern  simulations  used  to  quantify  total  training 
requirements. 

A  description  of  the  current  U&T  pattern  for  a  given  occupation  is  a  necessary  starting 
point  for  understanding  the  current  situation  and  for  developing  possible  management 
alternatives.  Appropriate  data  must  be  synthesized  from  a  variety  of  sources-most  notably 
from  TI/CODAP  occupational  survey  (OS)  data,  personnel  records,  training  management 
data  files,  the  training  catalogue  (AFR  50-5),  TDS  surveys,  and  informal  interactions  with 
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functional  managers,  training  staffs,  and  field  supervisors-to  gain  a  comprehensive 
perspective  on  the  occupational  area. 

A  key  element  in  defining  career  patterns  is  the  identification  of  jobs  within  an 
occupation.  In  occupational  analysis  terminology,  "job"  refers  to  a  group  of  related  positions 
in  which  many  of  the  same  tasks  are  performed;  "position"  refers  to  a  unique  set  of  tasks 
performed  by  one  person  (Shartle,  1959).  Each  occupation  includes  a  number  of  jobs  that 
vary  in  content  according  to  organizational  level,  assigned  functions,  equipment  operated  or 
maintained,  level  of  experience  of  personnel,  and  a  number  of  other  interrelated  factors 
(Driskill  &  Mitchell,  1979;  Vaughan  1978). 

For  the  TDS,  major  job  types  are  identified  using  standard  OA  methods  for  analysis  of 
data  collected  from  job  incumbents  (Archer,  1966;  Christal,  1974;  Christal  &  Weissmuller, 
1988;  Cragun  &  McCormick,  1967;  Mitchell,  Ward,  &  Driskill,  1989).  The  OA  job  types 
provide  a  realistic  foundation  for  describing  the  current  U&T  pattern  for  Air  Force  enlisted 
occupations,  although  some  issues  remain  regarding  the  level  of  analysis  most  appropriate 
for  the  TDS  (Yadrick,  et  al.,  1988).  OSR  data  files  are  reprocessed  to  create  more  concise 
job  descriptions  based  on  the  TMs  of  the  occupation.  The  CODAP  MODULE  programs  are 
used  to  create  a  new  occupational  data  base.  Generally,  four  indices  are  used  to  characterize 
the  association  of  jobs  and  task  modules: 

1.  the  sum,  across  people  and  tasks,  of  the  percent  time  spent 

performing  the  tasks  of  a  TM; 

2.  a  running  cumulation  of  the  summed  percent  time  spent  index; 

3.  the  average  percent  time  spent  per  task  in  the  TM;  and 

4.  the  average  percent  members  performing  across  TM  tasks. 

Such  job  descriptions,  using  TM-level  data  to  characterize  the  functions  and 
responsibilities  of  job  incumbents,  provide  a  very  concise  picture  of  total  job  content  These 
summary  descriptions  are  used  as  TDS  data  files  [more  complete  descriptions  can  be 
generated  showing  all  tasks  by  using  ASCII  CODAP  Modules  programs]. 

The  identification  of  all  the  training  courses  and  job  assignment  flows  within  an 
occupation  is  a  more  difficult  problem.  No  existing  single  source  provides  all  of  the  training 
data  needed;  no  present  sources  provide  suitable  information  on  job  flows,  and  no  promising 
method  of  coordinating  different  sources  is  available.  For  TDS  purposes,  such  information 
is  synthesized  from  various  sources  to  identify  relevant  courses,  FIDs,  PME  programs,  and 
occupation-unique  training  delivery  systems.  A  Job  and  Training  History  survey  is  developed 
for  use  with  a  representative  sample  of  experienced  job  incumbents  (and  a  random  sample 
of  first-job  personnel).  Incumbents  identify  their  present  and  previous  jobs,  and  list  dates  of 
attendance  for  all  training  programs;  they  can  also  write  in  additional  training  programs. 
These  data  are  used  to  estimate  rates  of  attendance  for  the  relevant  courses  for  each  job, 
attrition  rates,  and  assignment  flows  (average  length  of  assignment,  PME  course  attendance 
points,  etc.).  Graphic  flow  patterns  are  developed,  as  well  as  narrative  summaries.  Such 
displays  and  data  are  validated  with  subject-matter  experts  (SMEs)  in  subsequent  interviews 
or  meetings  (see  example  in  Figure  2.6). 
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Figure  2.6.  Example  Utilization  and  Training  Pattern  Flow  Diagram. 

Graphic  displays  provide  some  sense  of  the  flow  of  individuals  through  training  programs 
and  jobs  but  do  not  lend  themselves  to  summarising  the  various  types  of  quantitative 
information  involved,  such  as  the  number  of  individuals  entering  the  field  each  year,  or  the 
various  probabilities  of  reassignment  among  jobs,  attending  advanced  technical  training,  or 
participating  in  PME  courses  (Mitchell,  et  al.,  1988).  In  constructing  the  occupational  model, 
an  analyst  must  conceptualize  the  flow  of  personnel  through  these  various  training  stages  and 
sequences  of  jobs. 

Once  decisions  are  made  as  to  which  jobs  and  courses  to  include  in  the  model  of  an 
occupation,  and  which  categories  of  training  (Job-driven,  Time-driven,  or  lime  and 
Job-driven)  are  most  appropriate,  then  a  quantitative  flow  model  can  be  developed.  Such 
data  are  best  conceptualized  as  a  series  of  data  matrices  which  reflect  the  probabilities  of 
personnel  moving  from  training  programs  to  initial  jobs  or  from  one  job  to  another.  To 
estimate  every  training-to-job  and  job-to-job  probability  becomes  an  extremely  complex 
problem;  new  techniques  had  to  be  developed  to  simplify  the  problem  and  reduce  the 
number  of  estimates  which  had  to  be  made.  This  was  done  by  collecting  cases  into  a 
common  reassignment  pool  -  a  "Collect." 
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Handling  School-  &  Job-to-Collect  Transitions  (TRMJOB)  -  When  individuals  exit  a 
training  course  or  a  job,  they  are  collected  in  the  simulation  into  one  or  more  transition 
nodes,  to  simplify  estimation  of  job  assignment  and  advanced  training  probabilities.  These 
transition  nodes  are  called  "Collects";  they  are  simply  a  device  for  making  estimation  of 
assignment  probabilities  feasible.  Without  such  a  mechanism,  probability  estimates  would 
have  to  be  made  for  every  possible  path  between  jobs,  training  states,  and  exits,  which  is 
overwhelmingly  complex. 

To  specify  these  variable  paths  in  the  model,  a  data  file  is  needed  to  define  a  set  of 
tenure  time  ranges  and  identify  the  relevant  collects.  The  FRMJOB  file  reflects  which 
collect  node  individuals  might  go  to  (from  each  job  and  between-job  training  state)  at  each 
reassignment  point  (tenure  range).  The  file  has  a  row  for  each  job  and  between-job  training 
state  and  columns  for  the  collects.  In  essence,  this  file  is  a  mapping  for  the  flow  of  people 
(entities)  through  the  collect  nodes. 

Handling  Collect-to-Job  Probabilities  (TOJOBSi  -  The  third  type  of  transition  is  from 
each  collect  to  the  next  job  or  training  program,  or  to  exit  from  the  field.  By  using  this  type 
of  transition,  it  is  possible  to  estimate  the  probabilities  of  such  assignments  by  using  OSR  or 
JTHS  data.  The  data  are  summarized  for  the  specialty  in  the  form  of  a  data  matrix  having 
rows  for  each  job  and  exit  possibility  (leave  nodes).  Leave  nodes  represent  a  summation  of 
all  ways  of  exiting  the  specialty.  Columns  of  the  TOJOBS  matrix  represent  each  transition 
node  (collects).  Each  cell  entry  (data  field)  in  the  matrix  contains  the  probability  of  moving 
from  the  collect  to  a  job,  between-job  training  state,  or  exit  (Leave). 

Even  if  we  could  accurately  reconstruct  the  job  and  training  history  of  all  individuals  in 
an  AFS,  this  would  not  enable  us  to  test  alternative  ways  to  organize  the  jobs  and  training 
programs  or  to  evaluate  predicted  trends.  To  achieve  the  flexibility  needed  to  assess  the 
impact  of  proposed  changes  on  a  specialty,  we  need  a  dynamic  simulation  of  the  AFS  which 
can  be  modified  in  order  to  test  different  job  structures  or  different  training  configurations. 
Such  a  simulation  creates  job  and  training  histories  for  a  hypothetical  AFS  population  for  10 
to  20+  years  in  order  to  estimate  the  annual  training  requirements  (including  OJT)  for  the 
specialty.  By  modifying  data  files  or  input  parameters  (such  an  annual  entries),  we  can  see 
the  impact  of  such  changes  on  AFS  training  requirements  and  costs. 

U&T  Pattern  Simulation  (UTPSIMi  Program.  For  any  given  set  of  input  data  (Current 
or  some  Alternative  U&T  Pattern),  the  TDS  dynamic  simulation  program  calculates  how 
many  individuals  enter  the  system,  flow  through  various  training  and  job  programs,  and  exit 
the  specialty  (Mitchell,  et  al.,  1988;  Yadrick,  et  al.,  1988).  The  UTPSIM  produces  an  Output 
Entity  History  file  (OUTEHF)  containing  the  job  and  formal  training  history  of  each 
simulated  entity  in  the  career  field  population  and  summary  reports  indude  monthly  job  and 
training  flow  statistics.  Data  files  for  a  specialty  are  developed  from  a  number  of  sources, 
to  provide  as  accurate  a  picture  as  possible  of  the  present  jobs  and  training  programs  of  the 
AFS  (see  Yadrick,  et  al.,  1988).  UTPSIM  is  then  run  to  provide  a  basis  for  quantifying 
training  requirements  of  the  AFS  and  total  annual  training  costs  and  training  capadties  of 
representative  units  are  estimated  with  the  Resource/Cost  Subsystem  (RCS)  programs. 
Current  U&T  pattern  results  become  the  baseline  for  evaluating  any  proposed  changes. 
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Training  Proficiency  (TRNPRF'l  Program  -  The  TRNPRF  computes  the  total  amount  of 
OJT  needed  in  the  specialty  for  all  individuals  to  achieve  required  proficiency  on  the  task 
modules  (TMs)  involved  in  their  jobs.  The  TRNPRF  uses  the  Entity  History  File  (EHF)  as 
its  basic  data  file.  It  also  employs  the  job  identities  (JOBIDS)  file  for  the  specialty  and  a  run 
parameter  (RUNPAR)  file.  The  TRNPRF  program  is  designed  to  estimate  OJT 
requirements  (in  terms  of  needed  training  hours)  for  each  entity  in  the  EHF  by  accumulating, 
for  each  TM,  the  hours  in  each  training  setting  (classroom,  hand-on,  self  study,  and  OJT) 
received  by  the  entity  as  he  or  she  progresses  through  various  training  courses  and  jobs. 
Whenever  an  entity  enters  a  job,  after  all  within-job  training  has  been  accomplished  (such 
as  FTDs),  the  entity’s  proficiency  level  is  computed  for  all  TMs  involved  in  the  job.  These 
computations  are  done  using  the  running  total  training  hours  and  the  allocation  curves.  If 
the  entity  has  less  than  full  proficiency  on  an  assigned  TM,  then  OJT  is  required  for  that  TM. 
The  actual  number  of  additional  OJT  hours  required  for  full  proficiency  for  each 
job-required  TM  can  be  computed  using  these  allocation  curves). 

There  are,  however,  two  complications  on  this  kind  of  computation  in  the  TRNPRF.  One 
is  where  the  additional  training  hours  required  to  achieve  full  proficiency  exceed  the 
maximum  that  can  be  attained  by  OJT.  In  such  cases,  the  program  accumulates  the 
maximum  effective  hours  (the  program  also  keeps  a  record  on  the  minimum  proficiency  level 
achieved  on  each  TM  under  these  circumstances).  A  second  complication  is  where  the 
number  of  TM  training  hours  an  entity  has  received  exceeds  the  maximum  effective  hours 
(for  that  TM  and  setting);  in  this  case,  only  the  maximum  effective  hours  are  used  to 
compute  the  entity’s  proficiency. 

Such  calculations  are  made  for  every  entity  in  the  Entity  History  File  for  all  relevant 
TMs,  depending  on  their  jobs  and  the  courses  they  have  attended.  These  data  are  then 
accumulated  into  monthly  training  events.  Such  monthly  statistics  are  compiled  for  all  the 
monthly  periods  in  the  period  of  interest  (a  number  of  years)  and  an  annual  rate  computed. 
The  results  of  these  computations  are  included  the  FUS  Output  file  which  is  essentially  a 
quantitative  data  base  of  all  training  requirements  for  the  specialty. 

Given  successful  runs  of  the  UTPSIM  and  TRNPRF  programs,  their  output  files  become 
the  input  files  for  a  series  of  programs  used  in  the  RCS  to  compute  training  resource 
capacities  and  costs  by  site  or  program.  The  capacity  programs  will  be  used  to  evaluate 
whether  the  increased  OJT  can  be  handled  by  local  OJT  trainers  or  exceeds  the  capacity  of 
some  units. 

The  UTPSIM  and  TRNPRF  programs  provide  an  analyst  with  considerable  flexibility  for 
modeling  various  proposed  changes  to  the  jobs  and  the  training  programs  of  a  specialty.  The 
current  U&T  pattern  serves  as  the  baseline  against  which  any  new  U&T  pattern  or  change 
in  parameters  (such  as  input  flow)  can  be  assessed.  The  TRNPRF  quantifies  the  training 
requirements  for  any  U&T  pattern  at  a  very  detailed  level,  and  calculates  the  OJT 
requirements  to  support  that  pattern  for  every  job  and  all  relevant  TMs.  The  TRNPRF 
Output  File  then  becomes  the  primary  input  for  the  RCS  programs  in  calculating  training 
costs  and  capacities  of  representative  units. 
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2.2.23  Resource /Cost  Subsystem  (RCS1  Technologies 


The  RCS  was  developed  in  order  to  provide  TDS  three  distinct,  yet  interrelated,  analytic 
capabilities: 

1.  determination  of  the  types  and  amounts  of  resources  required  to  provide  training  on 

each  TM  in  each  training  setting,  and  estimation  of  the  amounts  of  those  resources 
available  for  use  in  providing  training  at  each  site; 

2.  assessment  of  the  capacities  of  various  sites  to  accommodate  different  volumes  of 

training  on  different  combinations  of  TMs  in  different  training  states,  where  a 
training  state  consists  of  a  set  of  specific  amounts  of  training  conducted  on  specific 
TMs  in  particular  training  settings;  and 

3.  estimation  of  the  variable  costs  that  must  be  incurred  in  providing  training  on 

each  TM  in  each  training  setting,  and  in  providing  particular  volumes  of  training 
in  specific  training  states. 

To  accomplish  these  objectives,  the  RCS  was  structured  into  three  analytic  components: 
a  Resource  Requirements  Component  (RRC),  a  Training  Capacity  Component  (TCC),  and 
a  Cost  Estimation  Component  (CEC).  These  components  use  input  files  from  the  TCS  and 
FUS;  compile  resource  requirements,  availability  and  cost  factor  data;  and  interact  with  one 
another  to  generate  resource  and  cost  estimates  (see  Rueter,  Feldsott,  &  Vaughan,  1989). 

Resource  Requirements  Component.  The  RRC  performs  five  data  development 
functions.  Specifically,  it  (a)  determines  the  specific  types  of  resources  required  to  perform 
training  on  each  TM  in  each  training  setting,  (b)  estimates  the  quantity  of  each  identified 
type  of  resource  required  for  training  each  TM  in  each  setting,  (c)  produces  compilations  of 
those  estimates  classified  on  the  basis  of  the  ways  in  which  the  corresponding  types  of 
resources  affect  variable  training  costs  and  training  capacities,  (d)  estimates  the  quantities 
of  those  resources  available  for  the  provision  of  training  at  various  actual  sites,  and  (e) 
delineates  an  appropriate  set  of  representative  sites  for  the  particular  specialty. 

Inputs  to  this  component  include:  TM  definitions  and  amounts  of  time  allocated  for 
training  the  various  TMs  in  different  training  settings  (from  the  TCS),  and  preliminary  lists 
of  resources  required  for  training  each  TM  in  each  setting  (collected  via  a  Training 
Requirements  Questionnaire  administered  to  school  and  field  trainers.  Based  on  these 
inputs,  an  analyst  identifies  representative  sites  and  develops  the  basic  data  for  use  in 
estimating  training  capacities  and  costs  within  the  other  two  RCS  components. 

Representative  sites  (TTC  courses,  FTD  courses,  operational  units)  are  identified  to 
account  for  important  locational  variations  in  travel  and  temporary  duty  (TDY)  costs,  training 
loads,  missions,  resource  availability,  and  other  factors.  The  use  of  representative  sites 
permits  more  economical  collection  of  cost  data  and  simplifies  comparisons  of  resource 
availability  and  requirements.  A  key  consideration  in  determining  resource  availability  is  the 
identification  of  the  minimum  resources  required  for  operational  duties  at  representative  sites 
since  operational  requirements  place  real  constraints  on  the  training  capability  of  units. 
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Training  Capacity  Component.  The  TCC  evaluates  the  capacities  of  various 
representative  sites  to  provide  training  in  appropriate  settings  on  different  combinations  of 
TMs  and  in  training  volumes  that  are  compatible  with  the  U&T  patterns  identified  in  the 
FUS.  Inputs  to  this  component  consist  of  the  following:  TM  combinations  and  training 
volumes  for  the  various  U&T  patterns  (from  the  FUS),  predicted  amounts  of  specific 
resources  required  for  the  provision  of  training  on  each  TM  in  each  training  setting  (in  the 
form  of  regression  equations  from  the  RRC),  and  availabilities  of  those  resources  for 
providing  training  at  each  representative  site.  Resource  availability  data  are  collected  in  a 
Training  Resources  Availability  survey  of  TTCs,  FTDs,  and  representative  field  units.  For 
dedicated  training  resources,  data  on  the  sharing  of  equipment  and  other  resources  must  be 
collected  since  sharing  has  a  potential  impact  on  training  capability,  and  may  vary  by  site; 
only  that  amount  of  usage  for  the  specialty  of  interest  can  be  considered  . 

An  analyst  develops  estimates  of  the  capacity  of  each  representative  site  to  accommodate 
various  combinations  of  TMs  and  training  loads,  and  identifies  any  resource  limitations  that 
constrain  representative  sites  from  accommodating  particular  U&T  patterns.  When  such 
constraints  are  encountered,  they  are  displayed  in  the  OJT  Capacity  Report  for  each  site  as 
'Trainees  Unsupportable". 

Inherent  in  this  process  is  evaluation  of  the  feasibility  of  resource  substitution,  the  impact 
of  training  load  on  training  quality,  and  the  possible  impact  of  training  on  mission 
performance  (where  a  training  deficit  exists,  then  resources  must  be  diverted  from  mission 
performance  to  provide  training,  or  the  unit  runs  the  risk  of  error  or  accident  which  may 
result  from  the  lack  of  complete  training).  Training  capacity  evaluations  use  statistical 
training  resource  requirement  functions  and  mathematical  programming  formulations  to 
assess  various  training  options. 

Cost  Estimation  Component  (CEO.  The  CEC  computes  total  annual  variable  costs  for 
providing  training  of  each  TM  in  each  training  setting  (assuming  all  required  resources  are 
available  in  sufficient  quantities),  and  then  compiles  the  cost  estimates  in  a  form  compatible 
with  the  estimates  developed  for  training  capacity.  Inputs  to  this  process  include:  estimated 
training  resource  requirements  (from  the  RRC),  training  states  (i.e.,  amounts  of  time 
allocated  to  specific  TMs  in  specific  settings)  and  associated  training  volumes  compatible  with 
various  U&T  patterns  (from  the  FUS),  and  unit  resource  cost  factors  from  external  Air  Force 
data  sources  (TDY  costs,  instructor  salary  levels,  costs  of  training  equipment  and  supplies, 
etc.).  By  applying  the  unit  cost  factors  to  the  estimated  training  resource  requirements  for 
the  specified  training  states  and  training  volumes,  this  component  estimates  the  variable  costs 
of  conducting  training  in  each  training  setting  and  for  each  specified  training  volume  in  the 
corresponding  training  state. 

Once  these  very  complex  basic  data  sets  have  been  developed,  they  must  be  synthesized 
and  processed  as  formatted  reports  in  such  a  way  as  to  be  useful  for  Air  Force  decision 
makers.  This  is  done  by  operation  of  the  RCS  components  and  data  files;  the  common 
starting  point  is  the  FUS  Output  File  (from  TRNPRF).  Multiple  processing  is  performed 
to  generate  training  hours  for  trainees,  trainers,  and  other  resources,  for  both  classroom  and 
OJT  requirements.  It  should  be  noted  that  the  capacity  and  costing  programs  of  the  RCS 
operate  in  parallel,  use  some  common  data  files  as  input,  and  use  some  unique  files  as  well. 
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The  major  products  of  the  RCS  are  data  files  and  reports;  these  are  organized  by 
representative  site,  training  setting,  and  job. 

Such  unit-level  data  are  also  aggregated  to  derive  estimates  for  the  total  AFS,  for 
MAJCOMs,  and  for  individual  bases  in  a  summary  section  of  the  report.  Separate  reports 
of  training  capacity  and  costs  are  generated  for  the  current  U&T  pattern  and  each  alternative 
U&T  pattern  to  create  multiple  RCS  output  files.  [For  more  details  on  the  RCS  and  its 
components,  see  Rueter  et  al.,  1989.]  RCS  data  files  serve  as  the  basis  for  comparing  the 
impact  of  various  suggested  AFS  changes,  and  for  generating  reports  to  respond  to  managers’ 
inquiries,  through  the  operations  of  the  Integration  and  Optimization  Subsystem. 

2.2.3  Integrated  Technologies  in  the  IQS 

The  IOS  ties  together  the  other  three  subsystems  into  one  overall  functional  system 
(Figure  2.7;  see  also  Figure  2.3,  page  16).  The  IOS  contains  mechanisms  which  enable 
system  modeling  and  optimization.  The  interconnections  of  the  subsystems  provide  the 
capacity  to  optimize  measures  derived  from  one  subsystem  relative  to  constraints  obtained 
from  the  others,  and  to  simultaneously  process  data  files  derived  from  different  subsystems. 
The  IOS  also  provides  the  interface  with  users;  that  is,  the  subsystem  receives  all  requests, 
calls  appropriate  data  from  the  other  subsystems  or  TDS  files,  and  creates  products  to  meet 
the  users’  needs.  In  all  of  its  functions,  the  IOS  governs  the  interaction  among  the  TDS 
subsystems  and  various  data  sources,  and  the  relationship  of  the  system  with  various  types 
of  users.  The  IOS  is  structured  to  perform  three  types  of  functions: 

2.2.3. 1  Modeling  Functions.  The  IOS  processes  information  and  data  files  from  the  TCS, 
FUS,  and  RCS  to  create  various  models  of  the  AFS  under  consideration.  The  basic  model 
for  the  AFS  is  the  current  U&T  pattern;  each  alternative  U&T  pattern  represents  some 
change  to  the  current  U&T  model.  This  approach  provides  maximum  flexibility  in  the  TDS, 
because  an  almost  infinite  number  of  possibilities  can  be  considered. 

The  modeling  functions  of  the  IOS  are  not  limited  to  examining  the  alternative  U&T 
patterns  formulated  in  the  FUS  (although  AFS  managers’  preferences  are  collected  only  for 
these  alternatives).  Rather,  IOS  modeling  provides  the  capability  to  change  any  input 
variable  for  any  program,  thus  permitting  examination  of  the  impact  of  such  change  on  the 
total  system.  For  example,  the  simplest  change  would  be  to  use  the  current  FUS  model  and 
raise  or  lower  the  number  of  personnel  entering  the  ABR  course  (modify  the  Trained 
Personnel  Requirement  or  TPR).  The  system  would  then  generate  reports  for  comparison 
with  the  data  from  a  baseline  current  U&T  pattern  run;  differences  in  values  (annual 
training  costs,  total  AFS  population  in  future  years,  etc.)  would  reflect  the  relative  impact  of 
the  change.  Another  type  of  change  would  be  to  change  the  content  of  some  course,  such 
as  the  ABR,  and  then  run  the  system  to  assess  the  changes  in  costs  and  total  OJT 
requirements  for  the  specialty. 

Any  major  proposed  change  in  a  specialty  should  be  dealt  with  as  a  formal  alternative 
U&T  pattern,  so  that  possible  consequences  can  be  examined  in  some  detail  and  data 
collected  which  describe  Air  Force  managers’  preferences  among  a  set  of  alternatives.  Some 
changes,  such  as  a  merger  of  several  AFSs  at  the  technician  level  (as  proposed  in  RIVET 
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Figure  2.7  The  Integrated  Training  Decision  System 


WORKFORCE),  may  exceed  present  system  capacities  (unless  a  new  OSR  is  accomplished 
using  a  redeveloped  task  list  covering  ail  specialties  involved,  or  needed  data  are  estimated 
in  some  other  way). 

Most  of  the  possible  changes  which  might  be  considered  for  an  AFS  can  be  modeled  by 
changing  input  parameters,  course  content,  job  content,  or  career  patterns  within  the  TDS. 
Such  changes  are  processed  as  modeling  runs  with  specified  values  of  selected  variables. 
Analysis  then  focuses  on  how  such  changes  impact  on  training  capacity  and  total  training 
costs.  The  training  capacity  reports  generated  in  this  process  will  also  highlight  any 
constraints  on  training  capability  in  terms  of  training  equipment,  instructor  availability,  or 
other  significant  problems. 

An  example  may  serve  to  illustrate  how  potential  changes  can  be  evaluated.  The 
problem  might  be  a  resource  constraint  in  conducting  OJT  in  some  units,  such  as  not  enough 
hours  when  a  piece  of  test  equipment-a  Weapons  Release  Control  System  (WRCS)  Analyzer 
in  the  Radar  and  Inertial  Navigation  Systems  Maintenance  specialty  (AFS  328X4)— is 
available  for  use  in  training.  One  approach  to  the  problem  would  be  to  move  training  of  that 
equipment  from  an  OJT  setting  to  a  formal  course  (an  FTD  or  the  basic  resident  course  at 
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the  TTC).  The  first  possibility  can  be  modeled  in  the  TDS  by  adding  enough  hours  to 
achieve  the  required  proficiency  (as  indicated  by  the  Allocation  Curve  for  the  WRCS  TM) 
to  the  resident  course.  A  second  model  would  be  to  add  hours  for  this  training  to  the  FTD 
course. 

The  results  of  these  analyses  are  shown  in  Figure  2.8,  along  with  data  from  the  current 
U&T  pattern  as  a  baseline  for  comparison.  It  should  be  noted  that  the  "Exceeded"  under 
the  Current  U&T  pattern  (first  column)  indicates  that  some  resource  constraint  exists  (in  this 
case,  an  equipment  availability  constraint).  Note  also  that  the  proposed  solution  of  adding 
the  required  WRCS  Analyzer  training  to  the  FTD  (third  column,  far  right)  did  not  solve  the 
problem;  OJT  capacity  is  still  exceeded.  Moving  the  training  to  the  resident  (second  column, 
middle)  course  does  appear  to  solve  the  constraint  problem,  but  at  an  additional  ABR  cost 
of  about  $20,000.  There  is,  however,  some  offsetting  reduction  in  OJT  costs  since  WRCS 
Analyzer  training  was  removed.  In  this  particular  example,  the  costs  calculate  out  to  be 
about  the  same  but  providing  the  training  in  the  ABR  does  not  exceed  OJT  capacity  of  units 
in  the  field.  Obviously,  this  is  a  feasible  and  practical  solution. 

The  point  here  is  that  once  a  problem  and  potential  solutions  have  been  identified,  the 
IOS  modeling  capability  can  be  used  to  translate  the  potential  solutions  into  modifications 
of  the  specialty  data  base  (the  current  U&T  files)  to  create  a  new  alternative  U&T  pattern. 
Each  relevant  data  file  is  systematically  examined  and,  where  needed,  modified  to  accurately 
portray  the  change.  The  TDS  software  is  then  employed  to  generate  products  for  each 
potential  solution  and  the  resulting  reports  can  be  compared  to  the  baseline  data  (current 
U&T  products)  by  Air  Force  decision  makers.  Short  summary  comparisons  can  then  be 
developed  to  highlight  the  changes  (as  in  Figure  2.8). 
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2.232  Optimization  Functions.  Given  the  almost  limitless  number  of  possible 
changes  which  might  be  studied,  a  TDS  analyst  or  functional  user  might  wish  to  take  another 
approach  to  assessing  specialty  training  changes.  This  approach  could  make  use  of  the  IOS 
optimization  software.  The  analyst  or  user  can  specify  an  objective  function  or  goal  (such 
as  minimization  of  OJT  cost  or  total  training  costs,  or  maximization  of  the  amount  of 
available  equipment,  etc.),  run  the  optimization  program,  and  examine  the  effects  on  the 
specialty  if  the  objective  function  is  maximized  or  minimized.  The  analyst  can  ask  "What  if’ 
questions;  for  example: 

What  is  the  impact  on  total  training  costs  if  we  minimize  initial 
resident  course  instruction? 

What  happens  to  specialty  jobs  (proficiency),  if  we  maximize  FI  D 
training  and  minimize  OJT?. 

What  is  the  impact  on  proficiency  acquisition  and  training  costs, 
if  we  have  a  10%  cut  in  new  recruits  entering  training? 

Some  potential  optimization  problems  may  become  visible  during  modeling  runs  of 
the  specialty  as  training  constraints  are  identified,  or  possible  new  models  may  be  suggested 
by  initial  optimization  runs.  Other  possible  optimizations  will  be  suggested  by  general  Air 
Force  trends,  such  as  budget  cuts  or  changing  operational  priorities.  In  some  cases,  these 
could  be  complex  problems  with  several  constraints  and  multiple  values  to  be  optimized. 

The  approach  taken  in  employing  optimization  algorithms  to  solve  maximization  or 
minimization  problems  in  the  TDS  is  to  employ  modular  data  bases  and  seek  solutions  at  the 
lowest  possible  level.  This  isolates  solutions  to  only  the  area  of  interest  and  has  some 
efficiency  in  terms  of  saving  computer  time.  Only  the  largest  optimization  problems,  such 
as  minimizing  total  specialty  training  costs,  would  require  employment  of  the  entire  TDS  data 
base.  More  limited  problems  can  thus  be  dealt  with  by  limited  program  runs. 

The  computer  programs  used  to  provide  optimization  runs  were  designed  so  that  they 
can  be  interfaced  with  TDS  data  bases  in  the  IOS.  A  detailed  review  of  such  software  is 
beyond  the  scope  of  this  overview  report.  For  details  of  constraints,  available  routines,  and 
procedures  for  using  such  software,  see  the  TDS  Procedural  Guide  (Vaughan,  Mitchell, 
Marshall,  Feldsott,  &  Rueter,  1988). 

2.23.3  User  Interface  Functions.  The  IOS  also  serves  as  an  interface  with  TDS  users. 
It  receives  all  requests,  calls  appropriate  data  from  the  other  subsystems,  and  creates 
products  to  meet  the  users’  needs.  In  its  present  configuration,  the  interface  is  handled 
through  on-line  computer  requests  (control  cards,  specification  of  variables,  creation  of 
required  data  files)  and  the  resulting  products  (data  files  and  reports). 
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2.3  Independent  Yet  Integrated  Technologies 

While  each  of  the  TDS  subsystems  and  its  required  technological  innovations  were 
developed  to  be  independent  operations,  they  were  developed  sequentially  so  that  new  types 
of  data  or  processes  in  one  subsystem  helped  to  pattern  the  technology  possible  in 
subsequent  subsystems.  For  example,  the  Task  Modules  developed  in  the  R&D  of  the  TCS 
served  as  the  basis  for  development  of  U&T  patterns  in  the  FUS  and  resource  requirements 
and  availability  surveying  in  the  RCS.  However,  the  specific  techniques  used  to  cluster  tasks 
into  TMs  (i.e.,  co-performance  clustering  using  CODAP)  are  not  critical  to  the  operation  of 
the  TCS  or  the  other  subsystems;  any  clustering  or  presorting  process  could  be  used  as  long 
as  the  resulting  TMs  had  the  desired  characteristics  (shared  skill  and  knowledge 
requirements;  some  efficiency  through  common  training). 

In  addition,  the  co-performance  task  clustering  technology  may  have  uses  and 
applications  which  are  not  necessarily  related  to  the  TDS.  Such  task  clustering  might  have 
a  number  of  potential  uses  and  included  task  clustering  in  the  major  revision  CODAP  project 
which  was  already  underway.  This  new  task  clustering  technology  is  presently  being  used 
operationally  by  the  USAFOMC  and  other  research  organizations  as  an  aide  in  identifying 
and  interpreting  job  types  within  occupations  (Phalen,  et  al.,  1987,  1988). 

Thus,  we  can  say  that  the  innovative  co-performance  clustering  technology  which  was 
developed  in  the  TCS  R&D  is  an  independent  technology  -  one  which  is  useful  in  the  TDS 
but  not  critical  to  the  success  of  the  system  (i.e.,  other  task  groupings  could  be  used)  and  is 
a  technology  which  has  potential  utility  for  other  applications  not  involved  with  the  TDS. 
The  same  can  be  said  of  other  technologies  developed  for  use  in  the  TDS  subsystems 
including  U&T  pattern  modeling,  the  U&TP  simulation,  the  training  proficiency  requirements 
analysis  (allocation  curves),  resource  requirements  and  availability  accounting,  technology  for 
assessing  unit  OJT  capabilities  at  representative  sites,  and  estimation  of  composite  training 
costs  and  capabilities  for  a  specialty.  Each  of  these  technologies  is  a  highly  useful  part  of  its 
subsystem  and  of  the  TDS  as  a  whole,  but  are,  for  the  most  part,  independent  technologies 
which  have  potential  in  a  variety  of  non-TDS  applications. 

To  give  a  second  example,  the  U&T  Pattern  modeling  with  its  more  dynamic 
portrayal  of  the  jobs  and  training  programs  over  the  span  of  a  10  to  20+  year  period  is  itself 
a  technological  innovation.  It  goes  beyond  the  static  picture  of  a  specialty  presently  used  in 
the  USAF  occupational  analysis  program  (Mitchell,  et  al.,  1988).  Thus,  it  has  potential  for 
application  in  normal  occupational  analysis  operations  quite  independent  of  its  present  use 
in  the  TDS.  Further,  it  is  not  critical  to  the  operation  of  the  FUS  per  se;  that  is,  other  ways 
of  modeling  the  specialty  could  be  used  as  long  as  they  were  useful  in  the  conceptualization 
of  the  occupation  and  aided  in  the  estimation  of  training-to-job,  job-to-training,  and 
job-to-job  probabilities.  Thus,  while  the  present  U&T  modeling  technology  is  an  integral  part 
of  the  TDS,  it  is  also  an  independent  technology  with  other  possible  applications. 

Most  of  the  "pieces"  of  technology  developed  in  the  TDS  R&D  were  designed  as 
independent  technologies.  This  was  done  so  that  as  other  new  technologies  were  developed, 
they  could  be  implemented  with  minimum  difficulty.  This  modular  approach  was  intentional; 
it  makes  the  TDS  a  very  flexible  system  which  can  remain  useful  as  techniques  and  major 
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training  decision  making  processes  change  (see  Mitchell,  et  al.,  1987). 

We  expect  that  such  innovative  methodological  and  technological  changes  will  also 
make  obvious  a  need  for  a  technologically-advanced  data  collection,  generation,  analysis,  and 
evaluation  capability.  Such  changes,  which  might  be  linked  to  broader  innovations  in 
computer  (PC)  administration  of  surveys,  automated  flow  of  survey  data  from  bases  to 
AFMPC  or  the  USAFOMC,  or  other  data  integration  mechanisms,  may  themselves  suggest 
alternative  methods  and  techniques  for  improving  the  TDS. 

To  make  good  decisions  about  the  training  needed  for  an  Air  Force  specialty  or 
system,  decision  makers  must  be  able  to  visualize  and  understand  the  jobs  and  training 
programs  of  the  specialty  or  weapon  system  under  consideration  and  its  technical  training  and 
Professional  Military  Education  implications.  To  assist  and  aid  in  such  decision  making,  the 
TDS  must  be  flexible  enough  to  incorporate  new  and  different  sources  of  information  or 
alternative  methodologies  for  accomplishing  its  functions.  If,  for  example,  new  cost 
accounting  procedures  for  determining  the  costs  involved  in  FTD  operations  were  developed 
which  proved  to  be  more  realistic  than  current  TDS  estimates,  then  data  from  the  new  FTD 
costing  method  should  be  incorporated  into  the  TDS. 

In  this  way,  the  flexible  TDS  design  can  be  continually  refined  and  updated  to  take 
advantage  of  new  technologies  as  they  are  developed.  Only  in  this  way  can  the  TDS  remain 
at  the  "cutting  edge"  of  the  various  scientific  areas  involved  and  maintain  its  utility  to  the  Air 
Force. 
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3.0  SCIENTIFIC  CONTRIBUTIONS  OF  THE  TDS  RESEARCH 

A  number  of  the  technologies  developed  in  the  course  of  the  exploratory  Training 
Decisions  System  research  and  development  project  utilized  state-of-the-art  techniques  and 
operational  concepts  from  a  number  of  technical  areas.  This  chapter  will  review  several  of 
these  technologies  in  light  of  the  present  state-of-the-art  with  the  view  of  explicating  what 
implications  the  TDS  R&D  results  have  for  the  future  of  the  relevant  technical  area 
(academic  disciplines). 

3.1  Top-Down  Structured  Systems  Design  &  Operations  Research 

The  initial  development  of  the  TDS  was  undertaken  within  the  conceptual  framework 
outlined  in  the  statement  of  work  (SOW).  The  SOW  specified  an  overall  system  architecture 
(noted  in  Figure  23),  but  provided  few  details  of  subsystem  operations  or  structure. 
Subsequent  to  the  initial  TDS  Progress  Review  (in  early  1984),  one  of  the  senior  AFHRL 
managers  pointed  out  the  need  for  a  detailed  preliminary  systems  design,  and  suggested  that 
sophisticated  methodologies  were  available  in  the  recent  Operations  Research/Systems 
Analysis  literature  which  might  be  useful. 

3.1.1  Top-Down  Structured  Analysis  System 

The  TDS  Preliminary  Design  was  developed  using  a  top-down  structured  systems 
design  approach  (Vaughan,  et  aL,  1984).  The  preliminary  design  identified  the  purpose  and 
potential  uses  of  the  TDS,  specified  the  subsystems  and  their  relationship  with  data  sources, 
and  identified  components  within  each  subsystem  to  accomplish  specific  required  functions 
(ibid.).  This  preliminary  design  was  conceptualized  as  a  flow  of  inputs  from  a  number  of 
sources,  processing  in  the  TDS  subsystems,  and  outputs  consisting  of  user  reports  and  data 
files.  It  was  based,  in  part,  on  successful  earlier  OJT  cost  analysis  and  capacity  modeling 
R&D  (Eisele,  Bell,  &  Laidlaw,  1978;  Rueter,  Bell,  &  Malloy,  1980),  as  well  as  on  SOW 
requirements  and  rational  design  considerations. 

In  1985,  a  reanalysis  of  the  system  requirements  was  undertaken  to  expedite  the 
development  of  a  TDS  Demonstrator.  This  microcomputer-based  demonstrator  was 
requested  by  AFHRL  to  help  communicate  possible  TDS  capabilities  and  functions  to 
potential  users  of  the  system.  In  the  development  of  the  demonstrator,  it  was  necessary  to 
specify  the  overall  systems  design  and  subsystem  specifications  at  a  greater  level  of  detail 
than  previously  had  been  required. 

3.13  Use  of  STRADIS  in  TDS  Design 

After  review  of  a  number  of  available  software  development  systems,  the  Structured 
Analysis  Design  of  Information  Systems  (STRADIS)  was  selected  to  assist  in  the  overall 
system  software  design.  STRADIS  is  a  top-down  structured  analysis  system  which  guides 
design  efforts  based  on  overall  program  objectives  and  parameters.  It  involves  multiple  levels 
of  design,  beginning  with  overall  system  goals,  to  insure  that  adequate  details  are  available 
for  software  designers  and  data  developers.  STRADIS  is  a  program  developed  within  the 
McDonnell  Douglas  Corporation  (MDC)  and  is  used  to  assist  in  systems  design  throughout 
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MDC.  While  it  includes  proprietary  software,  as  an  MDC  product  it  could  be  used  for  the 
TDS  project  without  fee  (i.e.,  at  no  additional  cost  to  the  government). 

A  three-level  design  was  developed  which  provided  for  an  overall  systems  design, 
specification  of  each  TDS  subsystem,  and  details  of  the  required  integration  of  subsystem 
data  files  and  outputs  into  reports  for  users  of  the  system  through  the 
Integration/Optimization  subsystem  executive  component  (see  Mitchell,  et  al.,  1985  [revision 
A,  September  1987]:  Figures  4.1  and  4.2,  and  Appendix  B).  This  more  comprehensive 
systems  design  and  specifications  were  used  in  the  later  development  of  the  proof-of-concept 
system.  While  some  details  of  the  final  system  varied  from  the  STRADIS  design,  this  was 
generally  the  result  of  problems  in  data  collection  or  because  of  innovative  new  technologies 
developed  later  during  the  R&D  effort;  the  overall  design  proved  to  be  sufficiently  detailed 
for  use  in  the  development  of  the  system  and  its  supporting  software. 

Thus,  state-of-the-art,  top-down  structured  analysis  methods  were  used  in  developing 
the  final  TDS  design.  They  proved  extremely  useful  in  guiding  the  work  of  creating  the 
subsystems  and  their  data  collection  and  files.  It  was  also  used  to  guide  the  development  of 
software  needed  to  support  an  operational  TDS.  Some  of  the  results  of  the  final  TDS  design 
and  issues  which  arose  in  the  TDS  R&D  have  implications  for  the  general  areas  of 
Operations  Research  and  Systems  Analysis. 

3.1.3  Implications  of  TDS  for  Operations  Research  &  Systems  Analysis 

The  final  TDS  was  successful  in  terms  of  integrating  very  divergent  data  from  a  myriad 
of  sources  and  in  processing  such  data  in  such  a  way  as  to  achieve  a  here-to-fore  impossible 
result--a  realistic  estimate  of  the  major  costs  of  Air  Force  on-the-job  training  programs  within 
a  specialty.  By  proper  formatting  of  data,  it  was  possible  to  integrate  occupational  analysis 
results  (task  and  job  data  and  patterns)  with  personnel  and  training  data  (personnel  flow 
rates;  training  flows  and  attrition  rates)  as  well  as  cost  data  (manpower  costs,  travel  and  per 
diem  rates,  resource  costs  and  availabilities)  to  achieve  the  desired  result  This  type  of  use 
of  totally  independent  data  sources  is  not  new  in  the  area  of  operations  research,  but  an 
application  in  the  area  of  quantifying  specialty  training  requirements  and  estimating  the  costs 
of  OJT  programs  is  clearly  innovative. 

The  implication  for  Operations  Research  and  Systems  Analysis  is  that  it  is  quite 
possible  to  merge  data  from  several  independent  technologies  to  create  new  and  useful 
applications.  The  need  for  this  kind  of  joint  technological  development  among  diverse 
academic  areas  will  probably  increase  in  the  future  as  the  success  of  this  type  of  innovative 
application  becomes  known  to  Human  Resources  and  MPT  managers  and  planners. 

A  second  implication  of  the  TDS  R&D  project  comes  from  the  creation  of  a  TDS 
demonstrator  to  simulate  operation  of  the  system.  This  was  a  very  useful  adjunct  for  the 
TDS  R&D  effort  in  that  it  permitted  potential  users  to  better  visualize  the  operation  of  a 
completed  system.  It  was  more  specifically  useful  in  letting  users  review  report  formats  and 
the  amounts  of  information  being  presented.  As  a  result  of  feedback  from  potential  users, 
data  report  formats  were  redesigned  and  several  new  types  of  displays  were  created  (for  the 
TDS;  they  have  not  yet  been  modified  in  the  demonstrator).  Thus,  in  any  R&D  effort,  it  may 
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be  useful  to  create  this  type  of  demonstrator,  to  properly  communicate  the  appearance  and 
functions  of  the  system  being  developed. 

3.2  TDS  Cost  Estimation  and  Cost  Accounting  Models 

The  main  conceptual  issue  that  had  to  be  resolved  in  developing  cost  estimation 
procedures  for  use  in  the  Resource/Cost  Subsystem  (RCS)  of  the  Training  Decisions  System 
(TDS)  was  the  proper  allocation  of  the  costs  of  shared  resources  among  the  different 
applications  in  which  they  are  utilized.  Resources  that  are  used  in  the  provision  of  Air  Force 
training  are  shared  among  different  applications  in  four  distinct  ways.  Specifically,  individual 
resources  may  be  shared: 

(1)  between  training  programs  and  operational  duties, 

(2)  among  training  programs  in  different  Air  Force 
specialties, 

(3)  among  training  on  different  task  modules  (TMs)  in 
individual  AFSs,  and 

(4)  between  group  instruction  and  training  of  individual 
personnel. 

In  developing  the  procedures  used  in  the  RCS  to  allocate  appropriate  portions  of  the 
costs  of  such  resources  to  specific  training  applications  (i.e.,  to  the  provision  of  training  on 
specific  TMs  within  specific  AFSs  in  specific  training  settings),  the  basic  principles  and 
techniques  of  activity-based  costing  have  been  applied. 

3.2.1  Activitv-Based  Costing 

Activity-based  costing  is  an  emerging  approach  to  cost  accounting  that  is  being 
adopted  by  increasing  numbers  of  manufacturing  companies.  The  fundamental  principles 
upon  which  it  is  based  are  that  "...activities  consume  resources  and  products  consume 
activities"  (Turney,  1989,  p.  25).  Turney  further  explains: 

Activities  are  procedures  or  processes  that  cause  work  ...The 
performance  of  these  activities  triggers  the  consumption  of 
resources  that  are  recorded  as  costs  in  the  accounts.  The 
activities  are  performed  in  response  to  the  need  to  design, 
produce,  and  distribute  products  (Turney,  1989:  25). 


Therefore,  in  allocating  the  costs  of  shared  resources  to  specific  products: 

Activity-based  costing  traces  costs  to  products  according  to  the 
activities  performed  on  them  (Turney,  1989:  23). 

To  apply  these  principles  to  estimation  of  the  costs  of  training  programs,  it  is  necessary 
merely  to  interpret  the  training  programs  as  products,  and  to  interpret  the  use  of  resources 
in  the  provision  of  group  instruction  or  individual  personnel  training  on  specific  TMs  in 
specific  training  settings  as  activities. 
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Activity-based  costing  explicitly  recognizes  that  the  costs  associated  with  different 
activities  can  originate  from  distinctly  different  basic  operations.  Accordingly,  in 
activity-based  costing,  the  costs  associated  with  different  activities  are  allocated  to  individual 
products  based  on  the  distinct  operations  that  drive  the  costs.  For  example,  in  a 
manufacturing  situation,  the  costs  involved  in  setting  up  a  production  run  are  first  traced  to 
the  specific  product  that  created  the  demand  for  the  setup  activities,  and  then  are 
apportioned  over  the  units  of  that  product  manufactured  in  that  production  run.  In  this 
manner,  low-volume  products  properly  incur  relatively  high  costs  per  unit  for  setup  activities, 
whereas  high-volume  products  incur  relatively  low  unit  setup  costs  (Turney,  1989:  25). 
Similarly,  in  the  RCS,  the  costs  associated  with  group  instruction  are  first  related  to  the  size 
of  the  group  receiving  the  instruction,  and  are  then  spread  over  the  term  of  the  training 
received  by  the  group,  thereby  deriving  an  appropriate  estimate  of  cost  per  student  week  for 
the  training  program. 

3.2.2  Comparison  to  Traditional  Costing 

In  contrast,  in  the  same  situations,  traditional  costing  would  allocate  setup  costs  among 
products,  and  group  instruction  costs  among  training  programs,  on  the  basis  of  measures 
related  to  total  volumes  of  production  and  training,such  as  total  direct  labor  hours  and  total 
direct  trainee  hours.  Traditional  costing  would,  therefore,  incorrectly  allocate  an  equal 
amount  of  setup  cost  to  each  direct  labor  horn  and  an  equal  amount  of  group  instruction  cost 
to  each  direct  trainee  hour,  regardless  of  the  length  of  the  production  run  in  which  the 
product  was  manufactured  and  the  size  of  the  group  to  which  the  training  program  was 
delivered.  As  a  result,  traditional  costing  would  generally  provide  distorted  estimates  of  the 
costs  of  individual  products  and  individual  training  programs.  Distortions  would  occur 
because  volume-related  cost  drivers  fail  to  trace  the  costs  of  volume-unrelated  activities 
correctly. 

Distorted  cost  estimates  inherently  involve  cross-subsidies,  wherein  one  product  or 
training  program  absorbs  costs  that  properly  should  be  assigned  to  another  product  or 
training  program  (Turney,  1989:  25).  More  importantly,  managerial  misinterpretations  of 
distorted  cost  estimates  as  accurate  measures  of  the  incremental  costs  of  individual  products 
or  training  programs  can  cause  incorrect  resource  allocation  decisions.  At  the  extreme,  as 
Kaplan  stresses: 

Seriously  distorted  product  costs  can  lead  managers  to  choose 
a  losing  competitive  strategy  by  deemphasizing  and  overpricing 
products  that  are  highly  profitable  and  by  expanding 
commitments  to  complex,  unprofitable  lines  (Kaplan,  1988:  64). 

Conversely,  as  Turney  states: 

...activity-based  costing  eliminates  cross-subsidy  by  using 
volume-unrelated  cost  drivers  to  trace  the  cost  of 
volume-unrelated  activities  to  the  product  (Turney,  1989: 25-26). 

For  example,  within  the  RCS,  cost  estimates  for  resources  used  in  group  instruction  are, 
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in  effect,  based  on  the  number  of  groups  entering  into  the  training  program,  rather  than  just 
the  number  of  personnel  involved  in  the  program. 

3.2.3  Development  of  an  Activity-Based  Costing  System 

A  good  activity-based  costing  system  should  account  for  pertinent  expenses  incurred 
throughout  the  entire  organization.  As  Kaplan  explains: 

All  company  resources  support  production  and  sales.  Even 
corporate  expenses  should  be  allocated  to  product  costs, 
especially  if  they  vary  across  product  lines  (Kaplan,  1988:  65). 

Thus,  an  effectual  costing  system  must  determine  how  indirect  costs  vary,  both  with 
regard  to  volume  and  with  regard  to  the  activities  associated  with  the  multiple  applications 
of  the  shared  resources.  Often,  extensive  allocations  of  the  costs  incurred  by  support 
departments  will  be  necessary  to  develop  estimates  of  the  costs  of  the  activities  performed 
by  the  departments  (Kaplan,  1988:  64). 

In  such  situations,  the  tracing  of  costs  from  resources  to  activities  and  then  from 
activities  to  applications  cannot  be  performed  with  great  precision.  Rather,  as  Cooper  and 
Kaplan  emphasize: 

...it  is  better  to  be  basically  correct  with  activity-  based  costing, 
say,  within  5%  or  10%  of  the  actual  demands  a  product  makes 
on  organizational  resources,  than  to  be  precisely  wrong  (perhaps 
by  as  much  as  200%)  using  outdated  allocation  techniques 
(Cooper  and  Kaplan,  1988:  100). 

3.2.3. 1  Subjectivity  -  The  cost  estimates  developed  by  activity-based  costing  systems  will 
also  be  more  subjective  than  those  derived  with  traditional  costing  methods  (Kaplan,  1988: 
64).  Cooper  and  Kaplan  explain: 

The  process  of  designing  and  implementing  an  activity-  based 
cost  system  for  support  departments  usually  begins  with 
interviews  of  the  department  heads.  The  interviews  yield 
insights  into  departmental  operations  and  into  the  factors  that 
trigger  departmental  activities.  Subsequent  analysis  traces  these 
activities  to  specific  products  (Cooper  and  Kaplan,  1988:  99). 

In  accord  with  these  precepts,  the  basic  approach  that  has  been  used  in  allocating  the 
costs  of  shared  resources  in  the  initial  implementation  of  the  RCS  has  involved,  first, 
interviewing  subject-matter  experts  (SMEs)  at  Air  Training  Command  (ATC)  headquarters, 
the  headquarters  of  the  operational  major  commands,  and  the  Air  Staff  to  determine  how 
Air  Force  training  is  organized,  and  how  resources  are  used  in  performing  that  training. 
Then,  questionnaires  have  been  distributed  to  a  larger  number  of  SMEs  to  obtain  their 
estimates  of  the  amounts,  or  proportions,  of  time  that  trainers,  trainees,  or  specific  types  of 
equipment  typically  are  or  should  be  involved  in  group  instruction  or  individual  personnel 
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training  on  particular  TMs  in  specific  training  settings.  Estimates  of  the  numbers  of  trainees 
who  currently  do  or  should  use  the  various  shared  resources  in  each  situation  have  also  been 
obtained  from  these  SMEs.  The  judgments  furnished  by  the  SMEs  have  then  been  used  as 
the  bases  for  allocating  appropriate  portions  of  the  corresponding  shared  resources  to 
individual  TMs. 

3 .2.3.2  Simplicity  -  Although  a  good  activity-based  costing  system  must  be 
comprehensive,  its  design  should  not  be  unnecessarily  complex.  The  cost  of  developing, 
operating,  and  maintaining  a  system  can  be  decreased  by  simplifying  its  design.  Therefore, 
a  well-designed  system  should  provide  no  more  detail  than  that  required  for  the  situation  in 
which  it  is  intended  to  operate  (Turney,  1989:  30-31). 

3 .2.3 .3  Adaptability  -  Unless  an  organization  makes  major  changes  in  its  structure, 
technology,  or  mode  of  operation,  it  usually  is  unnecessary  to  conduct  additional  interviews, 
surveys,  or  data  analyses  more  often  than  once  per  year  (Kaplan,  1988:  65).  Rather,  by 
combining  existing  data  with  reasoned  judgments  about  certain  salient  factors,  an 
organization  can  normally  develop  serviceable  estimates  of  the  costs  of  new  or  modified 
products  or  programs  by  assessing  their  demands  on  associated  activities  and  resources. 
Indeed,  this  capability  is  the  primary  reason  for  adopting  activity-based  costing  as  the  basis 
for  the  cost  analysis  procedures  implemented  in  the  RCS. 

3.2.4  Implications  of  TPS  for  Cost  Accounting  &  Econometrics 

The  TDS  uses  current  state-of-the-art  cost  accounting  and  econometric  modeling  for 
estimating  specialty  training  costs;  the  application  of  such  techniques  for  calculating 
on-the-job  training  costs  and  capacities  is  an  innovative  new  use  for  these  technical  areas. 
This  particular  approach  has  yielded  reasonable  estimates  of  OJT  costs  which  have  never 
previously  been  possible.  For  the  first  time,  such  "hidden”  training  costs  can  be  made  visible 
to  functional  managers,  training  staff  personnel,  and  commanders. 

One  of  the  implications  of  the  successful  TDS  R&D  project  for  the  cost  accounting  area 
is  in  the  TDS  RCS  costing  of  major  restructuring  alternatives.  This  involves  the  capability 
for  predicting  the  costs  of  major  changes  in  jobs  and  training  programs.  Traditional  cost 
accounting  has  generally  been  involved  in  assessing  present  costs  or,  at  the  most,  onfy 
estimating  costs  for  small,  incremental  changes. 

The  success  of  the  TDS  R&D  project  indicates  that  estimating  the  costs  for  major 
future  possibilities  or  configurations  can  be  accomplished  with  this  approach.  Thus, 
activity-based  cost  accounting  can  be  used  in  a  planning  mode  to  estimate  costs  for 
alternative  configurations,  which  considerably  extends  the  traditional  role  of  cost  accounting 
through  integration  with  econometric  modeling  and  MPT  planning. 

3.3  TDS  Allocation  Carves  and  Learning  Carves 

In  general,  learning  curves  are  mathematical  functions  which  translate  the  amount  of 
training  into  performance  level.  "Amount  of  training"  may  be  operationalized  as  number  of 
training  trials,  training  time,  or  in  other  ways.  Performance  level  has  been  operationalized 
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in  various  ways,  including  time-to-perform  and  quality  of 
performance,  measured  or  assessed  in  many  different  ways. 

3.3.1  TPS  Allocation  Curves  as  Learning  Curves 

As  discussed  earlier,  in  TDS  allocation  curves  relate  training 
time  to  proficiency  level.  Thus,  TDS  allocation  curves  may  be 
viewed  as  learning  curves,  and  the  scientific  findings  concerning 
learning  curves  are  directly  relevant.  Research  on  learning  curves 
has  been  done  in  at  least  three  academic  disciplines — experimental 
psychology,  education,  and  industrial  engineering.  Learning  curve 
research  in  industrial  engineering  has  been  concerned  with 
efficiency  improvements  in  a  manufacturing  process  related  to  amount 
of  experience  with  the  process.  These  research  results  are  not 
particularly  relevant  to  the  TDS  and  are  not  reviewed  here.  What 
is  relevant  is  that  by  using  allocation  curves  to  translate  training 
hours  in  various  settings  into  amounts  of  relative  proficiency,  we 
have,  for  the  first  time,  been  able  to  develop  a  system  for  dealing 
with  all  possible  combinations  of  trairing.  This  is  remarkably 
innovative,  and  permits  the  TDS  a  degree  of  flexibility  in 
considering  alternative  training  configurations  which  has  not  been 
possible  in  the  past  (nor  in  most  decision  support  systems) .  This 
is  made  possible  by  dealing  with  allocation  curves  as  learning 
curves . 

A  brief  review  is  presented  below  of  learning  curve  findings  in 
experimental  psychology  as  related  to  the  TDS. 

3.3.2  Learning  Curves  in  Experimental  Psychology 

Learning  has  been  one  of  the  major  topics  of  experimental 
psychology.  A  massive  literature  on  learning  curves  exists  in 
experimental  psychology.  Typical  studies  involve  learning  of 
extremely  small  and  structured  tasks,  such  as  in  paired  associate 
learning.  Learning  curves  typically  relate  numbers  of  learning 
trials  to  response  speed.  Anderson  (1980)  sums  up  the  findings  of 
this  whole  area  of  learning  curve  research: 

"In  fact,  virtually  every  study  of  skill 
acquisition  has  found  a  straight-line  function  on 
a  log-log  plot"  [227]. 

This  relationship  reflects  a  power  law: 

proficiency  =  a  *  (training  time) 

where  a  and  fe  are  model  parameters. 

This  function  has  two  key  features:  it  is  negatively  accelerating 
and  zero  training  time  produces  zero  proficiency.  TDS  allocation 
curves,  however,  have  a  somewhat  different  mathematical  function: 

proficiency  =  a  *  (training  time)  +  fe  *  (training  time) 

However,  this  function  produces  zero  proficiency  for  zero 
training  time.  Also,  if  fe  is  negative,  this  function  is  negatively 
accelerated.  As  a  matter  of  empirical  fact,  fe  is  negative 
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or  zero  for  all  existing  TDS  allocation  curves.  Thus,  TDS  allocation  curves  are  consistent 
with  the  experimental  psychology  learning  curve  findings. 

The  second  order  polynomial  functional  form  used  for  allocation  curves  permitted  a 
direct  test  of  the  negative  acceleration  feature,  which  the  power  law  function  does  not  Thus, 
the  polynomial  function  offers  advantages.  However,  it  would  be  worthwhile  in  future  TDS 
research  to  examine  the  power  law  function  for  allocation  curves. 

3 33  Application  in  the  TDS 

The  TDS  allocation  curves  are  used  to  translate  hours  of  training  time  for  each  TM 
among  various  training  settings.  This  provides  a  flexibility  to  the  system  which  enhances  its 
capability  for  use  in  evaluating  possible  alternative  configurations  of  training.  Given  the 
capability  to  equate  training  hours  and  gain  in  proficiency  among  various  training  settings, 
it  is  possible  to  consider  a  wide  range  of  possible  configurations  of  training  (and,  indeed,  to 
optimize  efficiency  in  terms  of  minimizing  training  costs).  This  use  of  learning  curve 
technology  appears  to  use  state-of-the-art  technology  in  a  unique  new  application;  for 
planning  future  changes  and  evaluating  such  possible  changes  in  terms  of  training  cost 
consequences. 

33.4  Implications  of  TDS  for  Experimental  and  Educational  Psychology 

The  use  of  allocation  curves  in  the  TDS  is  a  very  practical  and  effective  application  of 
experimental  results  to  help  solve  real  world  problems.  The  implication  of  the  success  of  the 
TDS  R&D  is  that  there  are  great  benefits  to  be  realized  from  developing  and  validating  the 
use  of  experimental  results  to  help  solve  real  world  problems.  TDS  use  of  allocation  curves 
also  demonstrates  the  value  of  merging  the  technologies  of  diverse  disciplines;  in  this  case, 
the  combination  of  using  Experimental  and  Educational  Psychology  research  (learning 
curves)  with  innovative  cost  accounting  predictions  to  assess  possible  future  changes  in  job 
and  training  programs. 

The  successful  use  of  allocation  curves  as  multiple  functions  relating  training  time  in 
various  settings  to  the  degree  of  relative  proficiency  achieved  demonstrates  that  by 
developing  a  very  flexible  system,  researchers  or  analysts  can  provide  decision  makers  with 
much  more  useful  tools  to  assist  in  their  decision  making  than  was  previously  thought 
possible.  In  the  past,  decisions  have  been  made  subjectively  for  lack  of  such  tools,  and  the 
impacts  of  such  decisions  could  not  be  evaluated  until  their  real  effects  became  obvious.  The 
TDS  project  has  demonstrated  that  such  decisions  can  indeed  be  modeled  and  that  even 
complex  issues  can  be  evaluated  through  quantitative  modeling  techniques. 

3.4  TDS  &  Industrial/Organizational  Psychology 

There  are  a  number  of  areas  where  the  TDS  R&D  effort  made  use  of  state-of-the-art 
research  results,  procedures,  and  operational  concepts  from  the  area  of  I/O  psychology. 
These  areas  include  task-based  job  (occupational)  analysis,  data  collection,  statistical  analysis 
techniques,  and  modeling. 
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3.4.1  Task  Clustering 


One  of  the  major  problems  in  the  design  of  TDS  was  how  to  best  utilize  the  massive 
amounts  of  task  data  generated  for  each  AFS  in  the  Air  Force  occupational  analysis  program. 
The  technological  challenge  was  to  find  a  systematic  way  to  systematically  reduce  the  data 
to  a  reasonable  number  of  categories  which  could  used  as  a  basis  for  describing  jobs  or 
training  programs  and  to  be  the  unit  of  analysis  for  costing  and  capacity  estimation.  The 
objective  was  to  create  groups  of  tasks  or  Task  Modules  (TMs)  which  share  common  skills 
and  knowledges  and  thus  represent  some  efficiency  in  being  trained  together  (Perrin,  et  al., 
1988). 

3.4.1.1  Statistical  Clustering  -  An  extensive  review  of  the  literature  concerning  statistical 
clustering  technologies  was  conducted  to  identify  candidate  techniques  for  task  clustering 
(Vaughan,  Yadrick,  Dunteman,  and  Clark,  1984:8).  While  there  are  a  variety  of  statistical 
techniques  available,  few  if  any,  had  previously  been  applied  to  the  clustering  of  tasks,  since 
task-based  job  analysis  is  largely  a  military  related  activity  (Mitchell,  1988).  Other  job 
analysis  approa  'ies  have  tended  to  use  factor  analysis  and  related  technologies  since  then- 
objectives  focus  on  the  identification  of  more  generic  dimensions  of  work  (Primoff  and  Fine, 
1988).  Among  the  clustering  methods  examined  were:  the  Q-Sort  (Stephenson,  1969)  which 
uses  subject-matter  experts  judgements;  medium  technological  methods  (Mulligan,  1981a  & 
b;  Morey,  Blashfield,  &  Skinner,  1983;  Mojena,  1977;  Kuiper  &  Fisher,  1975);  and  statistical 
algorithms  for  hierarchical  clustering  (Edelbrock,  1979;  Ward,  1963;  Ward  &  Hook,  1963). 
Such  studies  often  present  statistical  indices  of  clustering  accuracy,  including  the  Rand 
statistic  (Rand,  1971)  or  Kappa  (Edelbrock,  1979). 

As  a  result  of  the  literature  review,  two  basic  clustering  techniques  were  selected  for 
further  evaluation  (Vaughan,  Yadrick,  Dunteman,  &  Clark,  1984).  These  were  the  Ward  & 
Hook  algorithm  (Ward,  1963;  Ward  &  Hook,  1963)  and  the  average  linkage  algorithm,  both 
agglomerative  hierarchical  techniques.  These  start  with  n  separate  single-member  groups, 
then  form  n-1  groups,  etc.,  and  so  on,  until  all  cases  are  included  in  a  single  group.  An 
analyst  must  determine  the  intermediate  point  at  which  optimal  clusters  exist  The  Ward  & 
Hook  algorithm  attempts  to  minimize  the  variance  within  clusters,  while  the  average  linkage 
algorithm  attempts  to  minimize  the  difference  between  the  group  mean  and  any  group 
member. 

Another  issue  concerned  the  selection  of  the  distance  measure  to  be  used  in  clustering 
tasks;  a  number  of  such  measures  were  considered  including  Euclidean  distance,  city-block 
distance,  and  other  Mahalobian  distance  measures,  as  well  as  task  performance  overlap  (task 
co-  performance). 

After  extensive  field  evaluations  comparing  candidate  task  clustering  methodologies,  a 
procedure  for  the  development  of  TMs  was  developed  which  makes  use  of  both  statistical 
clustering  and  SME  judgements  (Perrin,  et  al.,  1986).  This  procedure  was  validated  with  two 
additional  specialties  and  was  found  to  be  realistic  and  cost  effective  (Perrin,  Mitchell,  & 
Knight,  1986).  The  final  recommended  procedure  is  one  which  combines  the  power  of 
statistical  clustering  with  the  specialty  judgements  of  SMEs,  thus  capturing  the  advantages 
of  both  methods  (Perrin,  et  al.,  1988;  Vaughan,  Mitchell,  et  al.,  1989). 
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3.4. 1.2  Task  Clustering  and  CODAP  -  In  the  process  of  developing  task  clustering 
procedures,  the  Comprehensive  Occupational  Data  Analysis  Programs  (CODAP)  were  used; 
the  task  data  to  be  used  for  clustering  resides  in  CODAP  files  and  the  Ward-Hook  Algorithm 
is  used  in  this  system.  By  transposing  the  typical  CODAP  case  file,  a  clustering  of  tasks 
across  cases  became  possible  using  existing  hierarchical  clustering  programs  (Perrin,  et  al., 
1986).  Interpretation  aids,  in  the  form  of  reports  of  background  data  were  not  available;  only 
a  K-Path  listing  of  task  statements  could  be  generated  for  this  purpose. 

Several  possible  improvements  to  facilitate  task  clustering  in  CODAP  were  evident  and 
were  discussed  with  the  AFHRL  task  scientist  responsible  for  CODAP  since  a  major  revision 
of  the  system  was  underway.  As  a  result  of  these  discussions,  several  major  innovations  were 
introduced  into  CODAP  including  a  transpose  program  and  a  new  cummulative  time-spent 
function  for  MODULE  reports.  These  innovations  were  extremely  useful  in  completing  the 
task  clustering  activities  required  for  the  four  AFSs  studied  in  the  TDS  R&D  project 

3.4.13  Subsequent  ASCII  CODAP  Developments  -  TDS  task  clustering  activities  were 
briefed  in  a  number  of  forums  to  potential  TDS  users,  occupational  analysts,  I/O 
psychologists,  and  others  (Perrin,  Vaughan,  et  al.,  1985;  Yadrick,  Vaughan,  et  aL,  1985; 
Vaughan,  Yadrick,  et  al.  1985;  Mitchell  and  Phalen,  1985;  Phalen  and  Mitchell,  1987).  These 
briefings  and  TDS  reports  clearly  demonstrated  some  of  the  potential  value  to  job  analysts 
and  others  of  clustering  tasks  into  task  modules.  As  a  result,  the  redevelopment  of  CODAP 
(into  ASCII  CODAP)  project  included  provision  for  further  refinement  of  this  new  technique 
and  creation  of  products  to  aid  analysts  interpret  hierarchical  task  clustering  diagrams. 

The  revised  ASCII  CODAP  system,  which  is  now  operational  at  AFHRL  and  the  USAF 
Occupational  Measurement  Center,  include  several  new  task  clustering  capabilities.  The 
system  has  added  various  other  ways  of  computing  co-performance  (raw  ratings,  time  spent, 
symmetrical  versus  asymmetrical,  geometric  mean,  etc).  In  fact,  any  sort  of  other  basis  for 
clustering  (such  as  with  task  difficulty,  etc.)  are  now  feasible  in  ASCII  CODAP  (Hand, 
Haynes,  and  Weissmuller,  1989).  Such  capabilities  go  far  beyond  what  was  originally  needed 
for  TDS  but  provide  a  more  general  utility  to  the  system. 

New  interpretive  programs  have  also  been  developed.  MODTYP  is  a  new  ASCII 
CODAP  program  to  assist  in  the  selection  of  appropriate  task  clusters  (Hand,  et  aL,  1989:26). 
TASSET  is  a  program  used  for  looking  at  the  most  representative  tasks  (most  discriminating 
tasks  of  a  task  cluster).  CORCAS  is  used  for  identifying  those  cases  most  representative  of 
a  task  cluster  (or  most  important);  it  automatically  orders  cases  in  terms  of  their 
discrimination  value  for  representativeness  to  task  clusters  (Ibid.). 

These  programs  are  now  fully  developed  and  are  undergoing  operational  testing 
(Phalen,  et  al.,  1989).  In  addition  new  extensions  of  this  task  clustering  technology  are  under 
development  including:  JOBMOD  for  relating  job  types  to  task  modules  (matrix  of 
relationships)  and  CLUSMAP  for  mapping  the  two-way  distribution  of  cases  and  task 
modules  in  terms  of  their  joint  distribution  of  time  spent  ratings  (Ibid.). 

3.4.1.4  Implications  for  Training  Decisions  Technology  -  The  very  rapid  development 
of  task  clustering  technology  since  the  TDS  R&D  use  in  developing  TMs  for  the  initial  four 
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TDS  specialties,  suggests  that  a  variety  of  new  methods  are  now  available  for  use  in  any 
further  TDS  R&D  efforts.  These  new  methods  and  techniques  should  be  operationally  tested 
in  the  new  TDS  R&D  project  through  their  application  with  additional  AFSs.  If  validated, 
these  new  techniques  could  save  considerable  time  and  effort  for  TDS  task  module 
development  efforts.  In  addition,  new  display  techniques,  such  as  CLUSMAP  noted  above, 
may  prove  beneficial  in  the  validation  and  review  of  OSR  job  types  for  use  in  the  TDS  and 
other  task-based  R&D  efforts. 

It  should  be  noted  that  the  task  clustering  developments  in  ASCII  CODAP  came  about 
in  part  because  of  the  needs  of  the  TDS  R&D.  Such  ASCII  CODAP  developments  in  task 
clustering,  however,  have  progressed  far  beyond  what  was  originally  suggested;  indeed,  they 
have  resulted  in  a  more  general  task  clustering  technology  which  can  be  applied  in  the 
on-going  occupational  analysis  program.  As  Tartell  has  recently  noted: 

...  There  are  too  many  tasks  in  occupations  and  too  few 
personnel  available  for  training  design.  As  a  partial  solution  to 
this  dilemma,  the  task  clustering  concept  allows  the  collapsing  of 
long  lists  of  tasks  into  a  much  smaller  number  of  clusters.  These 
task  clusters  have  the  advantage  of  being  based  in  occupational 
data  which  means  that  the  tasks  have  some  basis  for  being 
grouped  together.  The  logical  extension  of  task  clusters  is  cluster 
analysis  for  underlying  constructs.  This  approach  lessens  the 
worldoad  without  diminishing  the  utility  of  the  information. 

(Tartell,  1988:307). 

Thus,  this  TDS-inspired  interest  and  development  have  resulted  in  a  series  of  practical 
applications  for  the  task  cluster  or  task  module  approach  to  assessing  occupational  jobs  and 
training  programs.  This  technological  innovation  has  found  rapid  acceptance  in  the  OA 
arena  and  has  great  potential  utility  both  for  clarifying  the  identification  of  job  types  and  in 
improving  the  efficiency  of  training  development  (Ibid.). 

3.4.2  Utifeation_atid  Training  Pattern  Modeling 

A  second  area  where  the  TDS  R&D  made  use  of  state-of-the-art  technology  was  in  the 
area  of  job  typing  and  the  description  of  utilization  and  training  (U&T)  patterns  based  on 
occupational  analysis  data.  As  noted  in  Chapter  2,  the  TDS  is  designed  to  model  an  AFS 
in  such  a  way  as  to  make  calculation  of  the  annual  flow  of  people  through  jobs  and  training 
programs  possible  and,  in  turn  the  quantification  of  training  requirements,  including  OJT 
programs.  This  is  done  in  order  to  predict  the  costs  of  all  types  of  training  as  well  as  assess 
the  capacity  of  representative  field  units  to  support  OJT  (Vaughan,  et  al.,  1989). 

3.4.2. 1  Assignment  Probabilities  -  The  Field  Utilization  Subsystem  (FUS)  was  designed 
to  make  use  of  the  most  current  developments  in  occupational  analysis  (OA)  technology 
(Yadrick,  et  al.,  1988).  However,  the  present  OA  system  utilized  by  the  Air  Force  collects 
cross-sectional  data,  thus  developing  a  picture  of  the  jobs  and  training  of  an  AFS  at  a  given 
point  in  time  (Ruck,  1982;  Mitchell,  et  al.,  1987;  1988).  The  technological  challenge  for  the 
TDS  R&D  project  was  to  find  a  way  to  utilize  such  cross-sectional  data  to  estimate  job  flows 
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and  assignment  probabilities  in  a  dynamic  simulation  or  to  collect  new  data  which  would 
permit  creation  of  a  personnel  and  training  flow  model. 

This  was  done  successfully  in  the  TDS  R&D  project,  by  reanalyzing  OA  data  by  average 
assignment  period  (TAFMS  year  groups)  and  combining  the  static  OA  data  with  statistics 
from  a  Job  and  Training  History  survey  as  well  as  flow  statistics  from  ATC  and  the  Uniform 
Airman  Record  (UAR)  files.  By  using  a  modeling  mechanism  to  pool  all  reassignees  at  a 
particular  TAFMS  point  into  a  single  group  (a  "collect"  -  see  Chapter  2),  we  were  able  to  use 
percent  of  total  group  membership  as  an  assignment  probability,  thus  solving  one  of  the  most 
difficult  estimation  problems  in  the  project  (Yadrick,  et  al.,  1988;  Vaughan,  et  al.,  1989). 
This  made  possible  the  development  of  a  dynamic  simulation  of  AFSs  which  can  estimate 
annual  job  assignment  and  training  program  attendance  probabilities  with  a  reasonable 
degree  of  accuracy  (Yadrick,  et  al.,  1989;  Mitchell,  et  al.,  1987).  Given  this  technological 
breakthrough,  the  remaining  problems  in  modeling  of  AF  specialties  were  solvable  and  the 
TDS  became  a  reality  (Vaughan,  et  al,  1989).  Such  modeling  provides  the  flexibility  needed 
to  deal  not  only  with  the  present  AFS  status  (Current  U&T  Pattern)  but  any  reasonable 
other  AFS  structure  which  might  be  proposed  (Alternate  U&T). 

3.4.2.2  Implications  for  Occupational  Analysis  -  This  new  AFS  modeling  capability 
exceeds  the  present  technological  development  in  any  task-based  job  or  occupational  analysis 
system  (see  Gael  1988  for  a  comprehensive  review  of  this  technical  area).  The  ability  to 
construct  a  model  of  a  specialty  which  provides  a  more  dynamic  picture  of  the  jobs  and 
training  requirements  of  the  occupation  thus  represents  a  major  new  innovation  for  the 
occupational  analysis  technological  area. 

Although  the  TDS  was  developed  primarily  for  the  use  of  the  Air  Force  in  supporting 
training  decisions,  the  technological  breakthrough  in  TDS  modeling  of  AFS  U&T  patterns 
is  of  broader  scope  than  just  training  decisions.  This  technology  could  be  used  to  enhance 
the  reporting  of  AFS  data  in  the  normal  OA  process  for  other  uses,  particularly  for 
communicating  to  others  the  complex  jobs  and  training  programs  which  presently  exist  and 
for  examination  of  current  AFS  structures  and  re-  specification  of  responsibilities  (AFR 
39-1). 


Future  TDS  R&D  should  focus  on  identifying  other  potential  uses  for  this  kind  of 
occupational  modeling,  particularly  in  the  areas  of  multi-specialty  studies  and  the  new 
weapon  systems  acquisition  process  (see  the  TDS  Transition  Plan:  Vaughan,  et  al.,  1985). 
The  system  might  also  have  application  in  inter-service  studies  or  for  civilian  use. 

3.4.3  Data  Collection  Methods 

A  third  area  where  TDS  made  use  of  state-of-the-art  technologies  was  in  development 
and  use  of  data  collection  instruments.  Again,  the  types  of  surveys  done  in  the  TDS  R&D 
were  based  on  current  Air  Force  OA  technology,  but  went  beyond  the  present  state  of  this 
technology  by  collecting  information  from  more  carefully  selected  target  samples  and  by 
collecting  tailored  sets  of  information  from  targeted  groups  of  SMEs. 
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3.4.3. 1  TPS  Surveys  -  Allocation  surveys,  for  example,  which  would  have  been 
overwhelmingly  complex  if  one  survey  had  been  administered  to  all  SMEs,  were  divided  into 
shorter,  more  realistic  groupings  of  TMs;  these  were  administered  to  just  those  SMEs  known 
to  be  experienced  with  the  tasks  involved  (as  identified  by  OA  job  group  membership  from 
the  last  OSR).  This  approach  proved  quite  successful  (Perrin,  et  al.,  1988).  Likewise,  job 
and  training  history  surveys  were  targeted  to  individuals  from  the  last  OSR  assigned  to 
identified  job  groups  (plus  a  random  sample  of  recruits  with  six  months  to  two  years  in  their 
initial  assignments).  This  provided  a  more  realistic  sample  for  the  purpose  of  computing  job 
membership  and  job-based  school  attendance  than  would  have  been  possible  with  even  a 
much  large  random  sample  (Yadrick,  et  al.,  1988). 

Data  collection  for  the  Resource/Cost  Subsystem  surveys  used  much  smaller  samples, 
since  only  one  or  two  individuals  would  have  the  information  as  to  resource  requirements  for 
teaching  a  resident  course  or  a  specific  FI  D.  These  data  collection  efforts  also  required 
more  in  terms  of  the  amount  of  detail  needed  on  particular  types  of  training  resources,  how 
much  they  are  used  at  present,  and  their  availability  for  use  in  other  courses  or  AFSs  (the 
allocation  of  shared  equipment  or  resources).  In  the  case  of  the  RCS  surveys,  personalized 
administration  proved  a  necessity  in  order  to  standardize  instructions  and  answers  to 
respondents  questions.  Where  personal  administration  was  not  possible  (as  with  some 
remote  or  overseas  FTD  courses),  very  extensive  telephone  interaction  was  made  to  insure 
that  the  SMEs  understood  the  task  and  had  the  same  opportunity  to  have  their  questions 
answered.  There  were  some  inadequacies  of  the  information  gathered  through  this  process; 
future  work  should  focus  on  methods  to  insure  more  complete  data  for  RCS  use. 

3.4.3.2  Data  Collection  Methodological  Research  -  Where  there  was  any  question  as  to 
the  suitability  of  a  data  collection  method,  the  TDS  data  collection  effort  could  itself  be  used 
as  a  laboratory  to  assess  alternative  methodologies.  For  example,  while  it  has  long  been 
maintained  that  collecting  multiple  ratings  of  the  same  items  (tasks  or  task  modules)  on  the 
same  rating  form  creates  an  artificial  inflation  of  some  ratings,  there  was  very  little  formal 
literature  to  support  this  assertion.  The  allocation  rating  project  provided  an  excellent 
opportunity  to  test  this  assertion,  and  to  determine  which  data  collection  format  was  more 
realistic  for  TDS  use. 

An  experimental  design  was  developed  which  used  different  survey  formats  (multiple 
ratings  on  one  form  versus  ratings  on  separate  forms  in  the  same  booklet)  and  were 
administered  to  randomly  assigned  SMEs  in  the  same  AFS.  Analysis  of  results  indicated  that 
there  was  indeed  a  statistically  significant  artifactual  inflation  of  correlations  (.10  or  greater) 
between  scales  when  the  data  were  collected  on  a  single  rating  form  (Perrin,  et  al.,  1987). 
If  this  result  can  be  confirmed  by  other  OA  researchers  in  independent  studies,  it  has  veiy 
substantial  implications  for  future  OA  data  collection  efforts,  as  well  as  other  AFHRL  R&D 
projects.  Indeed,  the  result  may  have  implications  for  many  data  collection  instruments  in 
the  broader  I/O  psychology  area. 

3.4.4  Implications  of  TDS  for  Occupational  Analysis  and  I/O  Psychology 

Many  of  the  specific  technologies  developed  or  refined  in  the  TDS  R&D  project  might 
be  useful  to  the  existing  state-of-the-art  methods  of  occupational  analysis  in  particular  and 
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I/O  psychology  in  general.  The  modeling  of  an  occupation  done  in  the  TDS  could  be  very 
beneficial  to  OA  programs  as  a  tool  for  better  communication  on  the  variety  of  jobs  and 
training  programs  present  in  most  specialties.  Job  descriptions  using  TMs  are  more  concise 
that  complete  task  lists,  and  managers  can  more  quickly  grasp  the  similarities  and  differences 
that  exist  among  the  jobs  of  an  occupation  (AF  specialty).  The  U&T  pattern  simulation  is 
a  more  dynamic  representation  of  current  reality  in  a  specialty  than  the  present  static  displays 
of  job  membership  (Mitchell,  et  al.,  1987).  This  aspect  of  the  TDS  could  be  constructively 
implemented  in  the  operational  OA  program  of  the  Air  Force  (and  indeed,  of  all  the  military 
services)  today,  without  waiting  for  the  TDS  refinements  to  be  developed  in  the  next  R&D 
project.  Likewise,  our  findings  on  data  collection  formats,  targeted  sampling,  and  task 
clustering  could  also  be  used  immediately. 

The  task  clustering  programs  built  into  the  new  ASCII  CODAP  system  are  a  good 
example  of  immediate  implementation  of  a  TDS-inspired  technology  which  has  been  made 
operational  (Hand,  Haynes,  &  Weissmuller,  1989).  As  further  techniques  are  developed  and 
operationally  tested,  they  might  also  be  incrementally  implemented  (see  for  example,  the 
automated  job  and  task  cluster  programs  presently  being  tested;  Mitchell  and  Phalen,  1985; 
Phalen  and  Mitchell,  1987;  Mitchell,  Phalen,  Haynes,  &  Hand,  1988;  Phalen,  Staley,  & 
Mitchell,  1988). 

3.4.5  Implications  of  New  R&D  Technologies  for  the  TDS 

As  noted  earlier,  new  ASCII  CODAP  technologies  have  been  developed  and 
implemented  for  identifying  task  modules,  cases  characteristic  or  representative  of  task 
module  performance,  and  displaying  the  joint  performance  distribution  of  task  modules  and 
case  clusters  (jobs).  These  technological  improvements  occurred  after  the  TDS  analysis  of 
specialties  had  been  done,  and  thus  were  not  available  at  the  time.  However,  these  new 
technologies  should  be  used  in  any  future  TDS  studies,  to  make  use  of  new  state-of-the-art 
methods  (and  perhaps  to  provide  a  realistic  operational  test  and  evaluation  of  the  new 
procedures).  This  kind  of  testing  interaction  between  AFHRL  R&D  programs  can  provide 
considerable  benefits  to  both  programs,  and  to  the  Air  Force  MPT  community  as  a  whole. 

In  this  light,  we  need  to  be  aware  of  emerging  technologies  as  they  are  being  developed, 
in  order  to  make  optimum  use  of  new  methods.  For  example,  the  USAFOMC  has  requested 
research  into  interactive  survey  administration  techniques,  which  would  utilized  Air  Force 
personnel  information  networks  to  download  survey  instruments  to  base  level,  where  surveys 
would  be  completed  on  unit  microcomputers  (PCs).  If  this  technology  is  developed 
reasonably  soon,  it  might  be  employed  in  the  IDS  to  insure  more  timely  collection  of 
required  TCS,  FUS,  and  RCS  data,  as  well  for  enhancement  of  the  targeted  samples  needed 
for  some  types  of  information.  By  using  real-time,  operational  Air  Force  personnel  records 
to  locate  personnel  and  personnel  communications  networks  to  administer  surveys,  the  TDS 
data  collection  could  be  made  much  more  efficient  and  time  lost  in  mailing  time  and  failure 
of  sampling  (due  to  reassignments)  could  be  eliminated. 

This  is  but  a  single  example  of  possible  future  interaction  between  the  TDS  and  other 
AFHRL  R&D  programs,  where  the  use  and  testing  of  state-of-  the-art  new  methods  can 
benefit  both  programs.  Since  there  are  a  number  of  such  R&D  projects  currently  underway 
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which  deal  in  one  fashion  or  another  with  the  same  general  MPT  areas  as  are  addre: 1  ‘  in 
the  TDS,  there  are  a  substantial  number  of  potential  interactions  which  might  be  possible 
and  fruitful.  We  need  to  examine  some  of  these  possible  R&D  project  interactions  in  more 
detail;  several  are  discussed  in  some  detail  in  the  following  chapter. 
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4.0  POSSIBLE  INTERFACE  WITH  OTHER  R&D  PROJECTS 


Several  of  the  most  relevant  AFHRL  R&D  projects  which  might  have  a  significant 
interface  with  TDS  technology  are  discussed  below. 


4.1  The  Advanced  On-the-job  Training  System  (AOTS) 

The  AOTS  project  developed  a  computer-based  system  that  automates  job  site 
training  functions  and  integrated  the  related  management  functions  (Buescher,  Olvera,  & 
Besetsny,  1987).  The  initial  prototype  effort  has  completed  three  phases  of  design, 
development,  and  test  and  evaluation.  The  prototype  was  developed  at  Bergstrom  AFB,  TX 
to  support  some  of  the  jobs  in  the  Aircraft  Maintenance  and  Security  Police  career  fields. 
The  AOTS  R&D  project  was  designed  to  provide  incremental  products,  including  training 
requirements  definition  processes,  task  proficiency  evaluation  procedures,  training  resources 
identification  procedures,  and  automated  training  records  (Ibid). 

4.1.1  Basic  Components  of  AOTS 

The  AOTS  has  five  major  subsystems  which  address  OJT  management,  evaluation, 
curriculum  development  and  delivery,  computer  support,  and  personnel  support  AOTS  is 
a  task-  and  job-based  system.  A  master  task  list  for  the  specialty  is  used  as  a  basis  for 
specifying  tasks  to  be  trained;  a  number  of  generic  positions  (jobs)  are  recognized  which 
involve  the  performance  of  somewhat  different  sets  of  tasks.  In  addition,  individual  OJT 
supervisors  may  identify  additional  tasks  from  the  master  task  list  or  may  specify  new  tasks 
required  for  a  single  position  [new  tasks  are  theoretically  fed  back  to  the  USAF 
Occupational  Measurement  Center  (OMC)  for  inclusion  in  future  Task  Inventories].  The 
generic  position  task  list  plus  any  unique  or  new  tasks  (minus  any  task  training  received  in 
a  formal  course,  such  as  the  ABR)  are  the  tasks  which  the  OJT  supervisor/trainer  will  train 
a  given  individual.  Task  performance  will  be  tested  and  a  complete  task-by-task  record  of 
training  is  created.  This  training  record  will  follow  the  individual  airman  throughout  his  or 
her  career. 

4.1.2  Interface  with  the  TDS 

Since  both  AOTS  and  TDS  are  concerned  with  the  tasks  and  jobs  of  individual 
specialties,  they  generally  use  a  common  starting  point-the  AFS  task  list  developed  by  the 
USAFOMC.  In  the  AOTS  R&D,  the  OMC  task  list  was  the  starting  point  for  development 
of  the  AFS  Master  Task  List  However,  where  the  OA  program  and  TDS  are  concerned 
with  identification  of  the  major  variations  in  the  jobs  of  an  AFS,  the  AOTS  R&D  effort 
focused  on  the  specific  content  of  jobs  located  at  one  base  (one  major  command,  one 
mission,  etc.),  since  that  is  the  issue  for  OJT  trainers. 
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Fortuitously,  TDS  researchers  were  able  to  help  identify  generic  positions  and  generate 
initial  job  descriptions  from  OSR  data  for  use  by  AOTS  developers  in  developing  their 
generic  position  descriptions.  Given  the  available  data,  some  job  descriptions  were  specific 
to  the  base;  others  were  more  generic  (based  on  similar  missions  and  organizations).  AOTS 
SMEs  refined  these  job  descriptions  and  used  them  as  a  basis  for  developing  OJT 
instruction,  performance  tests,  and  their  records  and  management  system. 

Since  the  TDS  and  AOTS  have  a  common  starting  point  (OSR  data  on  tasks  and  jobs 
of  a  specialty),  they  have  much  more  potential  for  interface  than  most  R&D  programs. 
Ideally,  the  TDS  could  be  used  to  identify  the  major  jobs  (generic  positions  in  AOTS  terms) 
and  would  provide  information  on  the  task  (or  task  module)  content  of  formal  training 
programs  for  the  initial  set  up  phase  of  AOTS.  The  AFS  current  U&T  model  of  the  TDS 
could  serve  well  for  guiding  needed  development  in  creating  the  AOTS  data  base.  Indeed, 
for  the  prototype  R&D,  TDS  researchers  retrieved  CODAP  data  files  for  the  AOTS 
specialties  and  reprocessed  data  where  needed;  the  relevant  CODAP  data  was  exported 
from  the  AFHRL  UNISYS  to  the  VAX  computer  equipment  supporting  AOTS. 

One  major  difference  in  TDS  and  AOTS  is  their  different  level  of  analysis;  TDS 
operates  at  the  Task  Module  (TM)  level  where  AOTS  presently  manages  OJT  instruction, 
testing,  and  records-keeping  at  the  task  level  (which  becomes  very  complex  for  highly 
technical  or  diverse  specialties,  which  can  have  500  to  2000  tasks).  Given  the  very  large 
number  of  jobs  and  tasks  involved  for  all  the  positions  (in  all  AFSs)  at  a  base,  it  might  be 
wise  to  explore  the  possibility  of  managing  OJT  in  AOTS  at  the  TM  level.  If  the  tasks  of 
TMs  do  capture  the  same  skills  and  knowledges  (as  asserted  in  the  TDS  R&D)  this  would 
simplify  records-keeping,  expedite  development  of  OJT  instruction  and  testing,  and  ease  the 
burden  of  detail  on  OJT  trainers  and  supervisors.  [Note:  the  initial  AOTS  design  included 
provision  for  capturing  the  identity  of  which  tasks  belonged  to  which  TMs  for  each  AFS. 
This  design  feature  would  expedite  development  of  TM-level  management  of  AOTS  task 
module  training.] 

As  AOTS  is  implemented  operationally  and  data  files  compiled,  much  of  the  AOTS 
data  could  be  extremely  useful  for  future  TDS  studies.  Since  AOTS  will  have  specific  data 
on  trainer  and  trainee  time  spent  on  task  training,  that  data  can  be  summated  to  the  TM 
level  to  validate  and  refine  the  training  time  estimates  now  used  in  the  TDS.  Additionally, 
AOTS  would  eventually  generate  data  on  training  resource  requirements  and  availability  at 
a  level  of  detail  which  also  would  be  useful  in  the  TDS  [At  present,  the  TDS  would  be  able 
to  estimate  which  resources  are  used  in  such  training  and  which  are  critical  as  training 
constraints].  Likewise,  AOTS  data,  summated  across  types  of  bases  and  missions,  would 
greatly  refine  TDS  identification  of  representative  sites,  the  jobs  involved  in  each  such  site, 
and  verify  job  task  (TM)  content.  Conversely,  the  TDS  system  costing  of  OJT  could  serve 
as  the  initial  AOTS  costing  system  until  more  specific  cost  accounting  procedures  can  be 
developed  for  AOTS.  Once  developed,  such  AOTS  cost  accounting  results  could  validate 
TDS  estimates  of  OJT  or  be  used  as  refined  inputs  for  computing  total  AFS  training  costs. 
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The  TDS  is  able  to  model  changes  in  tasks,  jobs,  and  training  programs  of  an  AFS,  and 
to  predict  the  cost  consequences  and  training  constraints  resulting  from  the  proposed 
changes.  This  would  provide  AOTS  with  a  much  quicker  adaptation  and  update  capability 
since  any  changes  (in  tasks,  jobs,  and  training  programs)  would  be  directly  and  immediately 
implementable  for  OJT  in  a  compatible  data  base. 

These  two  R&D  efforts  have  much  to  offer  each  other,  if  the  potential  interfaces  are 
developed  systematically  over  the  life  of  both  projects.  This  will  require  considerably  more 
interaction  between  R&D  staffs  and  project  managers  than  was  previously  necessary  (or 
possible).  Since  both  projects  are  managed  by  AFHRL/ID,  such  future  systematic  interface 
is  both  possible  and  highly  likely,  once  the  potential  benefits  of  such  interaction  are 
demonstrated  and  verified. 

4.1.3  Recommendation 

The  potential  interfaces  between  AOTS  and  TDS  should  be  developed  systematically 
and  aggressively  in  any  future  R&D  efforts.  The  potential  management  of  AOTS  at  the 
Task  Module  level  (vice  task  level)  should  be  explored  as  soon  as  possible  to  make  AOTS 
management  of  OJT  simpler  and  less  expensive.  The  potential  use  of  AOTS  data  to 
validate  and  refine  TDS  predictions  of  OJT  costs,  selection  of  representative  bases, 
quantification  of  training  resource  requirements  and  availabilities,  and  other  factors  should 
be  programmed  as  a  requirement  of  all  future  AFHRL  R&D  projects  in  both  areas.  The 
potential  use  of  TDS  in  the  training  requirements  analysis  phase  of  AOTS  projects  (i.e.,  use 
of  TDS  AFS  models  and  data  on  proposed  AFS  changes)  should  be  an  AOTS  R&D 
contract  specification. 


42  The  Job  Performance  Measurement  System  (JPMS) 

In  response  to  a  Congressional  request  to  validate  selection  tests  against  job 
performance,  the  DOD  established  a  Joint  Service  Committee  on  Performance 
Measurement  (JSCPM).  The  Air  Force  role  in  this  effort  has  been  the  development  of  a 
task-specific  Job  Performance  Measurement  System  (JPMS).  Primary  objectives  of  this 
research  have  been  to  construct  job  performance  measures  that  are  as  free  as  possible  from 
measurement  bias  and  do  so  based  on  a  conceptual  model  of  job  performance  in  the  Air 
Force  (Dickinson,  1986;  Kavanaugh,  Borman,  Hedge,  &  Gould,  1985,  1987). 

In  the  Air  Force  JPMS,  the  basic  job  performance  criterion  consist  of  measures  of  task 
performance;  other  JPMS  measures  under  study  include  a  Walk-Through  Performance  Test 
(WTPT),  four  rating  forms  to  be  completed  by  incumbents,  peers,  and  supervisors,  and 
several  more  global  job  dimension  or  job  worth  rating  forms.  The  hands-on  measures 
compromise  the  criterion  against  which  die  other  measures  are  being  evaluated  as  surrogate 
indices  of  performance  (Bierstedt,  1985). 
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4.2.1  Job  and  Task  Assessment 


The  WTPT  focuses  at  the  job  level  and  several  WTPTs  are  required  for  each  specialty 
(for  each  type  of  jet  engine,  for  example,  in  the  Jet  Engine  Mechanic  career  ladder).  The 
test  consists  of  two  parts;  a  hands-on  test  of  task  performance  and  an  interview  test  of  the 
steps  involved  in  performing  the  tasks.  Tasks  tested  were  selected  based  on  occupational 
analysis  data  under  a  systematic  task  selection  plan  (Alba,  et  al,  1985).  Tasks  were  selected 
for  each  of  four  specialties  to  provide  coverage  of  some  tasks  commonly  performed  by  most 
incumbents,  and  other  tasks  specific  to  significant  within-spedalty  job  groups.  Both  the 
hands-on  and  interview  test  items  were  developed  based  on  a  detailed  task  analysis  of 
on-the-job  performance  objectives  derived  in  field  visit  observations  and  interviews  with 
senior  technicians  at  representative  bases.  Such  data  were  translated  into  step-by-step 
performance  measures  for  each  task.  The  tests  were  administered  in  the  job  location  by 
third-party  evaluators  who  are  not  only  subject  matter  specialists  in  the  field  but  who  were 
also  intensively  trained  on  rating  procedures  and  the  standards  of  successful  performance 
for  each  step.  In  addition  to  scoring  incumbents  on  each  step  of  the  task,  the  evaluators 
also  record  the  amount  of  time  it  takes  each  incumbent  to  complete  each  task  (Driskill, 
Mitchell,  &  Ballentine,  1985). 

Preliminary  analysis  of  results  indicated  a  components  of  variance  interrater  agreement 
of  .90  for  evaluators  of  Jet  Engine  Mechanics  performance  (Bierstedt  1985).  In  addition, 
a  repeated  measures  ANOVA  indicated  no  differences  among  raters  although  there  were 
significant  differences  among  incumbents  on  performance,  as  expected.  Analyses  of  other 
measures  are  still  in  progress. 


Procedures  developed  in  this  research  effort  will  be  used  in  the  validation  of  the 
present  Armed  Service  Vocational  Aptitude  Battery  (ASVAB)  against  job  performance  and 
technical  school  performance.  This  major  research  effort  represents  a  methodological 
breakthrough  in  the  area  of  criterion  measurement,  and  the  job  performance  measures 
developed  have  considerable  potential  for  use  in  other  Air  Force  MPT  research  efforts. 


4.2.2  Job  .Performance.  As  A  Criteria  For  Evaluating  Training  E 
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One  of  the  areas  where  Air  Force  JPMS  data  could  be  useful  is  the  area  of  evaluating 
the  effectiveness  of  USAF  training  programs.  This  possibility  was  examined  in  a  conceptual 
study  during  the  summer  &  fall  of  1985  (Driskill,  Mitchell,  &  Ballentine,  1985).  These 
authors  concluded  that  with  some  modification,  JPMS  data  would  be  extremely  valuable  as 
a  criterion  for  a  variety  of  training  evaluation  uses.  In  addition,  they  developed  a  training 
evaluation  model  based  on  the  job  performance  model  of  Kavanaugh,  Borman,  Hedge,  & 
Gould  (1985,  1987).  Such  models  may  serve  as  a  basis  for  innovative  new  research  efforts 
in  assessing  the  impact  of  various  training  programs  on  job  performance  and,  ultimately, 
their  utility  in  terms  of  combat  readiness.  Such  research  would  require,  however,  a 
considerably  more  detailed  knowledge  of  tasks  and  how  they  are  accomplished  in  the 
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operational  environment. 

4.2.3  Job  Performance  Measurement  in  AOTS  and  TPS 


The  techniques  used  to  measure  task  performance  in  the  JPMS  project  were  applied 
to  a  large  degree  in  the  development  of  task  qualification  assessment  in  the  AOTS  project. 
AOTS,  however,  requires  development  of  task  performance  tests  for  every  task  in  the  AFS, 
where  JPMS  measures  are  only  a  sampling  of  AFS  tasks.  Given  the  eventual  creation  of 
task  performance  measures  for  all  AFS  tasks  in  AOTS,  this  would  greatly  facilitate 
development  of  sample  task  measures  for  JPMS,  and  make  development  of  the  more 
detailed  JPMS  measures  more  cost-effective. 

The  use  of  JPMS  technology  in  AOTS  task  measurement  development  is  an  excellent 
example  of  technology  transfer  between  AFHRL  R&D  projects.  Indeed,  in  this  case,  some 
of  the  personnel  originally  involved  in  JPMS  procedural  development  later  worked  on  the 
AOTS  project. 

In  the  IDS,  there  is  presently  no  role  for  job  performance  measurement,  since 
performance  assessment  is  not  considered  in  the  AFS  model  or  in  the  costing  system. 
However,  since  task  measurement  is  a  real  component  of  OJT  programs  (as  demonstrated 
in  the  AOTS),  the  costs  of  such  task  assessment  are  an  additional  cost  element  which  could 
and  should  be  included  in  TDS  cost  estimates  for  AFS  training.  As  the  costs  of  task 
performance  measures  become  visible  in  the  AOTS,  they  should  be  added  to  TDS  cost 
models. 

In  addition,  if  JPMS  measurements  are  eventually  implemented  to  assess  training 
effectiveness  (as  recommended  in  Driskill,  Mitchell,  and  Ballentine,  1985),  then  these 
measures  may  also  be  relevant  for  TDS.  TDS  could  be  used  to  model  alternative  training 
strategies  for  those  areas  where  JPMS  measurement  suggest  ineffective  training.  ATC  or 
AFHRL  could  also  experimentally  vary  training  methods  (formal  school  versus  FTD  versus 
OJT)  and  evaluate  these  alternatives  not  only  in  terms  of  training  effectiveness  but  also  in 
terms  of  total  training  costs  and  OJT  capacity  constraints,  through  the  TDS.  Specific 
experimental  designs  and  methodologies  would  need  to  be  carefully  designed  for  such  an 
"experimental  school  house." 

43  Small  Unit  Maintenance  Manpower  Analysis  (SUMMA)  System 

Small  Unit  Maintenance  Manpower  Analyses  (SUMMA)  is  a  system  of  models  for 
optimizing  aircraft  maintenance  task  and  specialty  allocations.  The  SUMMA  system  has 
been  described  in  detail  by  Moore,  Wilson,  Seman,  Eckstrand,  Lamb,  and  Lindeman  (1987); 
Lamb,  Eckstrand,  Seman,  and  lindeman  (1987);  Wilson,  Faucheux,  Gray,  Wilson,  Lamb, 
and  George  (1987);  and  Lamb,  Eckstrand,  Seman,  Lindeman,  Faucheux,  Gray,  Wilson,  and 
Boyle  (1988).  SUMMA  is  intended  to  be  a  tool  for  "identifying  and  evaluating  alternatives 
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and  for  selecting  ways  to  improve  the  Air  Forced  enlisted 
maintenance  task/ specialty  structure  (Lamb,  et  al,  1988)  .  SUMMA 
" integrate [ s ]  manpower,  personnel,  and  training  issues  in  a  single 
analytical,  integrated  system  of  models  (Lamb,  et  al,  1988)".  As 
these  quotes  suggest,  the  overall  purpose  of  SUMMA  is  to  help 
determine  the  best  AFS  structure  to  support  aircraft  maintenance, 
or  the  best  assignment  of  maintenance  tasks  to  AFSs. 

A  major  motivation  for  the  SUMMA  development  is  dispersed 
basing  of  aircraft.  Currently,  large  numbers  of  aircraft  (i.e., 
entire  fighter  wings)  are  located  at  single  bases.  In  contrast, 
under  dispersed  basing,  a  fighter  wing  would  be  spread  out  across 
many  small  operating  locations  so  that  only  a  few  airplanes  are  at 
any  particular  location.  Currently,  aircraft  maintenance  AFSs  tend 
to  be  very  specialized  with  respect  to  aircraft  systems  (i.e., 
328X4,  Avionic  Inertial  and  Radar  Navigation  Systems  Maintenance) . 
As  a  consequence,  many  AFSs,  and,  thus,  many  airmen,  are  required 
to  maintain  a  single  aircraft.  When  many  aircraft  are  at  a  single 
location,  this  type  of  AFS  structure  permits  efficient  manpower 
use.  However,  under  the  dispersed  basing  concept,  this  very 
system-  specialized  structure  requires  many  maintenance  personnel 
at  each  of  the  many  aircraft  locations  and  a  much  larger  total 
number  of  maintenance  personnel.  Furthermore,  maintenance 
personnel  at  dispersed  locations  would  probably  not  be  fully  used; 
the  current  AFS  structure  does  not  permit  efficient  use  of 
maintenance  personnel  under  dispersed  basing.  SUMMA  is  a  tool  for 
determining  an  AFS  structure  that  permits  more  efficient  manpower 
use  under  dispersed  basing,  while  being  acceptable  relative  to 
manpower,  personnel,  and  training  (MPT)  considerations  and 
constraints. 

4.3.1  Structure  and  Function 

SUMMA  has  six  major  components.  As  with  the  TDS,  the  SUMMA 
components  are  concerned  with  data  collection,  data  analysis, 
modeling,  and  optimization.  These  six  components  are: 

1.  Operational  and  Maintenance  Scenarios — The  first  step  of  a 
SUMMA  study  involves  defining  basing  plans,  missions,  aircraft,  and 
related  operational  features.  Maintenance  features,  such  as 
non-manpower  support  resources  available,  and  maintenance  and 
manpower  utilization  policies  are  also  defined  for  each  scenario  to 
be  studied. 

2.  Task  Identification  and  Analysis — Lists  of  maintenance  tasks 
are  developed  for  each  operational  and  maintenance  scenario. 
Candidate  tasks  are  formed  by  pairing  each  of  seven  standard 
maintenance  action  verbs  with  each  equipment  item  on  the  subject 
aircraft  type.  These  task  candidates  are  then  screened  to  eliminate 
meaningless  and  irrelevant  tasks  and  to  reduce  the  total  number  of 
tasks.  Non-hardware  oriented  and  support  equipment  tasks  are  added. 
Next,  scores  are  obtained  for  each  selected  task  on  24  task  factors, 
including  such  factors  as  level  of  difficulty,  how  skill  is 
acquired,  and  how  long  can  this  skill  be  retained  without  practice. 
Subject-matter  experts  (SME)  provide  the  task  factor  ratings  in 
structured  interviews. 
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3.  Manpower,  Personnel,  and  Training  Data-In  this  component,  data  are  gathered 
concerning  the  current  AFSs  to  be  analyzed.  Eleven  categories  of  data  are  gathered, 
including  (a)  general  and  demographic  variables,  (b)  requirements  in  10  general  knowledge 
areas,  (c)  education  and  experience  requirements,  (d)  minimum  ASVAB  scores,  (e)  physical 
requirements,  (f)  training  course  characteristics,  (g)  special  pay  availabilities,  (h)  manning 
and  rotation,  (i)  labor  use  rates,  (j)  skill  upgrade  times  and  costs,  and  (k)  miscellaneous  AFS 
requirements  (i.e.,  security  requirements).  For  the  most  part,  these  data  are  obtained  from 
published  sources  and  from  the  Occupational  Research  Data  Bank  (ORDB).  Some  are 
obtained  from  SME  interviews. 

4.  Statistical  Analysis-In  this  component,  the  data  gathered  in  the  previous  three 
components  are  analyzed  to  form  the  SUMMA  data  base.  Two  analysis  types  are 
conducted-descriptive  and  cluster  analysis.  The  descriptive  analysis  is  primarily  interrater 
agreement  analysis  of  the  SME  ratings.  The  result  is  a  set  of  profiles  on  appropriate 
variables  for  each  task  and  AFS  in  the  SUMMA  study.  In  the  cluster  analysis,  tasks  are 
clustered  based  on  (in  some  cases  restructured)  scores  on  the  24  task  factors  gathered  in  the 
second  phase.  The  purpose  is  to  find  groups  of  tasks  which  might  be  new  AFSs.  The  Ward 
clustering  algorithm  is  used. 

5.  Decision  Support  System-The  SUMMA  Decision  Support  System  (DSS)  is  a  set  of 
models  and  optimization  analyses  which  use  the  data  from  the  first  four  components,  along 
with  user  inputs  to  find  a  best  allocation  of  tasks  to  candidate  AFSs.  The  DSS  includes  four 
types  of  models/optimizations:  (a)  a  task  allocation  optimization  model,  (b)  a  detailed  work 
flow  model,  (c)  an  MPT  evaluation  model,  and  (d)  a  cost  evaluation  model.  The  task 
allocation  optimization  model  uses  mathematical  programming  to  assign  tasks  to  AFSs  so  as 
to  minimize  aircraft  down  time  and  maximize  manpower  use  rates,  subject  to  constraints. 
The  detailed  work  flow  model  both  provides  needed  data  to  the  task  allocation  model  and 
is  used  to  verify  task  allocation  model  results.  The  Air  Force’s  Logistics  Composite  Model 
(LCOM)  is  used  for  detailed  work  flow  modeling  in  SUMMA  SUMMA  contains  an 
interface  to  LCOM.  The  MPT  evaluation  model  provides  detailed  analysis  of  MPT  variables 
for  an  AFS  structure,  such  as  that  developed  by  the  task  allocation  model.  Outputs  from  the 
MPT  evaluation  model  include  (a)  manpower  and  manpower  requirements  statistics,  (b) 
training  pipeline  manpower  requirements,  (c)  recruiting  statistics,  (d)  training  length  forecast, 
(e)  Permanent  Change-of-Station  (PCS)  move  forecast,  (f)  On-the-  job  training  (OJT) 
projection.  The  training  length  forecast  computes  the  number  of  weeks  in  each  AFS  required 
for  training  in  an  Airman  Basic  Resident  (ABR)  course,  in  field  training  detachments  (FTDs) 
and  in  basic  training.  The  OJT  projection  analyzes  the  three-skill-level  to  five-skill-  level 
upgrade  process,  including  numbers  of  OJT  hours  required  (based  on  data  gathered  in  the 
second  component)  and  numbers  of  weeks  required  to  upgrade,  taking  into  account  the  AFS 
manpower  analysis.  The  cost  evaluation  model  uses  results  from  the  MPT  evaluation  model 
to  estimate  the  total  basic  military  costs  (basic  pay  and  allowances)  for  the  new  AFSs, 
training  costs  for  formal  training,  PCS  costs,  and  recruiting  costs. 
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6.  Evaluation  of  Solutions--This  component  allows  users  to  conduct  sensitivity  analyses 
and  other  evaluations  of  the  DSS  results  and  to  generally  modify  and  rerun  various  SUMMA 
analyses. 

4 32  Comparison  to  TPS 

Both  SUMMA  and  TDS  are  MPT  models  designed  to  assist  in  identifying  and  evaluating 
alternative  AFS  structures.  However,  the  two  systems  differ  in  many  ways: 

1.  SUMMA  is  primarily  driven  by  mission  and  manpower  considerations  -its  primarily 
purpose  is  to  find  new  AFS  structures  which  improve  sortie  and  manpower  use  rates.  In 
contrast,  the  TDS  is  primarily  training  oriented-its  main  purpose  is  to  evaluate  MPT 
structures  with  respect  to  training  impacts.  The  TDS  focuses  primarily  on  MPT  structure 
within  an  AFS,  while  SUMMA  contains  little  detail  concerning  within  AFS  structure  and  is 
concerned  with  between-AFS  structure. 

2.  SUMMA  is  weapons  system-oriented.  It  considers  most  or  all  AFSs  that  support  a 
particular  weapons  system,  but  only  those  tasks  in  an  AFS  the  relate  to  the  specified  weapons 
system.  In  contrast,  the  IDS  is  AFS-oriented.  The  TDS  analyzes  entire  AFSs,  including  all 
weapons  systems  in  an  AFS,  but  typically  examines  only  one  or  a  few  AFSs  in  a  study. 
Furthermore,  the  TDS  is  not  restricted  to  maintenance  or  systems-oriented  AFSs. 

3.  SUMMA’s  stated  purpose  is  to  provide  analysis  in  all  three  of  the  MPT  areas,  while 
TDS  focuses  primarily  on  training.  The  two  systems  are,  in  fact,  much  more  similar  in  this 
respect  than  might  appear.  This  is  because  a  key  part  of  the  integrated  TDS  model  is  a 
manpower  model-the  FUS  U&T  pattern  model.  The  real  similarities  and  differences 
between  SUMMA  and  the  TDS  in  this  respect  can  be  seen  by  examining  each  of  the  MPT 
areas  in  more  detail. 

3a.  Manpower-Both  SUMMA  and  the  TDS  contain  manpower  models.  In 
SUMMA,  manpower  modeling  is  done  in  the  task  allocation  model,  LCOM,  and  the 
MPT  evaluation  model.  In  TDS,  the  U&T  pattern  simulation  is  a  manpower  model. 
Both  systems  do  manpower  modeling  and  analysis,  but  the  systems  focus  on  different 
aspects  of  manpower.  SUMMA  derives  manning  requirements  and  use  rates  from 
mission  requirements  and  operations/maintenance  scenarios.  In  contrast,  TDS  models 
task  performance  requirements,  but  does  not  relate  those  to  missions  or  use  rates.  The 
TDS  model  the  flow  of  people  through  their  entire  careers,  including  job  changes  and 
training  courses  taken  along  the  way.  In  contrast,  SUMMA  does  not  model  career 
paths,  except  one  standard  first  job  career  path,  and  ignores  job  differences.  SUMMA 
treats  all  people  at  a  particular  skill  level  as  the  same. 

3b.  Personnel-SUMMA  contains  some  personnel  modeling  features,  although 
rather  crude  ones.  TDS  contains  no  explicit  personnel  modeling.  Personnel 
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characteristics  of  an  AFS  (i.e..  aptitude  scores)  are  assumed  to  be  constant  and  to  be 
reflected  in  other  TDS  data. 

3c.  Training-SUMMA  contains  training  modeling  features.  These  features  allow 
training  costs  to  be  estimated  for  the  three-  to  five-skill-level  upgrade  process,  assuming 
no  significant  changes  in  training  structure  except  those  driven  by  a  new  AFS  structure. 
For  example,  SUMMA  assumes  that  tasks  trained  in  an  ABR  course  would  continue  to 
be  trained  to  the  same  degree,  although  which  tasks  are  covered  in  which  AFS’s  ABR 
course  might  change.  SUMMA  does  estimate  pipeline  manpower  and  calendar  time  for 
skill  upgrading,  considering  other  manpower  use.  The  TDS  contains  detailed  training 
modeling  features.  In  contrast  with  SUMMA,  the  TDS  focuses  on  position  qualification 
training,  rather  than  skill  upgrade  training.  The  TDS  considers  all  previous  formal  and 
on-the-job  training  to  estimate  OJT  quantities  required  for  position  qualification  at  each 
job  change.  The  TDS  is  designed  to  study  changed  training  structures.  It  permits  trades 
to  be  examined  among  various  training  delivery  methods,  physical  locations,  course 
structures,  and  course  attendees.  The  TDS  considers  student  labor  costs,  instructor  labor 
costs,  and  travel  and  per  diem  costs.  The  TDS  estimates  both  labor  and  nonlabor 
training  resource  requirements  and  permits  training  capacity  analysis.  However,  the 
TDS  cannot  estimate  training  resource  availabilities  as  a  function  of  manpower 
availability  and  mission  requirements,  as  SUMMA  can. 

4.  SUMMA  and  TDS  have  many  similarities  in  the  data  gathering  and  analysis 
methods.  For  example,  both  systems  use  the  same  statistical  clustering  algorithm  to  find 
task  groups.  However,  different  data  are  used  for  this  clustering  and  the  task  groups 
have  different  purposes  in  the  two  systems. 

4.3.3  Recommendations 

We  believe  that  SUMMA  and  TDS,  although  different,  can  be  highly  complementary. 
Each  attempts  to  take  an  integrated  approach  to  MPT  analysis,  although  for  different  stated 
purposes.  Each  contains  detailed  modeling  features  not  contained  by  the  other.  In 
manpower,  SUMMA  explicitly  derives  manpower  requirements  from  mission  requirements. 
The  TDS  does  not.  However,  the  TDS  explicitly  models  job  differences  and  changes  and 
career  flows,  while  SUMMA  does  not.  SUMMA  contains  explicit  personnel  modeling 
features,  while  the  TDS  does  not.  SUMMA  contains  relatively  undetailed  training  analysis 
compared  to  the  TDS.  TDS  permits  detailed  trade  analysis  on  training  structures,  but  does 
not  estimate  training  pipeline  manpower  requirements  or  impacts  on  OJT  of  changing 
mission  requirements.  We  believe  TDS  and  SUMMA  features  could  be  integrated  into  a 
single  model.  Furthermore,  certain  features,  such  as  more  sophisticated  personnel  modeling, 
which  neither  TDS  nor  SUMMA  have  could  be  put  in  the  integrated  model. 
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4.4  The  Integrated  Simulation  Evaluation  Model  Prototype  (ISEM-P) 

The  Integrated  Simulation  Evaluation  Model  Prototype  (ISEM-P)  is  a  computer 
program,  written  in  the  SIMSCRIPT  II.5  language,  which  simulates  the  basic  planning 
activities  and  decision-making  procedures  involved  in  the  Air  Force  Manpower  and 
Personnel  System  (AFMPS).  It  contains  a  dynamic  modular  representation  of  the  AFMPS 
in  which  aggregate  manpower  planning,  training  program  management,  detailed  personnel 
assignment  scheduling,  and  actual  personnel  flows  are  characterized  as  integrated, 
interdependent  activities  that  determine  the  status  of  groups  of  Air  Force  personnel,  and 
thereby  control  the  evolution  of  the  force  structure  over  time  (see  Rueter,  et  al.,  1981). 
ISEM-P  has  been  designed  as  a  prototype  for  a  large-scale  model  that  would  serve  as  an 
analytic  tool  for  predicting  and  evaluating  the  impacts  of  changes  in  policies,  procedures,  and 
environmental  conditions  on  the  performance  of  the  entire  AFMPS. 

4.4.1  Overview  of  the  Air  Force  Manpower  and  Personnel  System  (AFMPS1 

The  AFMPS  is  responsible  for  the  procurement,  development,  maintenance,  and 
deployment  of  the  human  resources  available  to  the  Air  Force.  Thus,  it  exercises  substantial 
control  over  the  fundamental  characteristics  of  occupational  life  in  the  Air  Force:  where 
people  are  stationed,  what  jobs  they  perform,  and  what  training  they  receive.  The  AFMPS 
also  influences  entry  into  and  exit  from  the  service  through  the  actions  it  takes  at  the  control 
points  associated  with  airman  and  officer  procurement  and  retention.  Through  these 
decisions,  it  provides  the  essential  link  between  people  and  jobs  that  enables  the  Air  Force 
to  accomplish  the  objectives  established  in  the  Five-Year  Defense  Plan  (FYDP). 

The  AFMPS  is  operationally  partitioned  into  three  components:  manpower,  personnel, 
and  training.  The  manpower  component  is  responsible  for  determining  the  effective 
demands  for  skilled  personnel  in  various  occupational  specialties,  at  various  levels  of 
expertise  and  authority.  The  personnel  component  is  responsible  for  maintaining  adequate 
supplies  of  personnel  who  have  suitable  skills  and  experience,  and  who  are  appropriately 
located  geographically  to  satisfy  the  demands  defined  by  the  manpower  component.  The 
training  component  is  responsible  for  recruiting  and  developing  personnel  to  replenish  or 
augment  existing  supplies,  and  furnishing  the  education  that  enables  current  personnel  to 
acquire  new  skills  and  become  more  proficient  at  old  ones.  Through  such  actions,  a  dynamic 
mechanism  is  provided  for  reconciling  skilled  personnel  supplies  and  demands  over  time. 

4.4.2  Comparison  of  TPS  and  ISEM-P  as  Models  of  the  AFMPS 

Obviously,  the  three  components  of  the  AFMPS  are  interrelated,  and  their  performance 
can  be  highly  interdependent.  Modification  of  the  procedures  employed  by  any  one  of  the 
components  can  have  substantial  effects  on  the  ability  of  the  other  components  to  fulfill  their 
responsibilities.  Thus,  it  is  important  to  incorporate  essential  features  of  all  three 
components  within  any  model  intended  to  aid  decision-making  relating  to  the  AFMPS. 
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In  this  regard,  the  Training  Decisions  System  (TDS)  focuses  on  the  training  component 
of  the  AFMPS,  and  models  that  component  in  considerable  detail.  In  particular,  the  TDS 
develops  specific  training  programs  expressly  designed  to  prepare  individual  people  for  the 
specific  jobs  to  which  they  are  assigned  as  they  progress  through  careers  within  a  single  Air 
Force  Specialty  (AFS).  The  manpower  and  personnel  components  of  the  AFMPS,  however, 
are  represented  only  implicitly  in  the  TDS,  within  the  transition  probabilities  describing  the 
flows  of  personnel  between  jobs  in  the  specialty. 

In  contrast,  ISEM-P  contains  more  explicit  representations  of  Air  Force  policies  and 
procedures  affecting  the  size  and  composition  of  the  stock  of  personnel  and  the  movements 
of  personnel  between  assignments.  It  includes  numerous  specialties  and  substantial 
geographic  detail.  It  operates  at  a  much  higher  level  of  aggregation  than  the  TDS,  however, 
with  regard  to  specialties,  people,  jobs,  and  training  programs. 

4.4.3  ISEM-P  Structure 

ISEM-P  is  an  aggregate  model.  It  deals  with  groups  of  personnel  rather  than 
individuals.  Every  personnel  group  is  characterized  by  one  or  more  of  five  attributes:  skill, 
grade,  year  group,  base,  and  time-on-station. 

Skill  is  designated  by  a  code  indicating  the  type  of  job  a  group  of  personnel  is  capable 
of  performing.  ISEM-P  skill  codes  correspond  fundamentally  to  the  AFSs  used  by  the 
AFMPS  to  categorize  occupational  expertise.  However,  most  ISEM-P  skill  codes  constitute 
mergers  of  several  AFSs.  Altogether,  there  are  51  airman  skill  codes  and  40  officer  skill 
codes  included  in  ISEM-P.  Within  each  skill,  personnel  are  stratified  by  grade,  or  rank.  As 
with  the  skill  codes,  an  ISEM-P  grade  usually  represents  more  than  one  actual  Air  Force 
grade.  ISEM-P  includes  five  grades  for  airmen  and  five  grades  for  officers. 

A  year  group  specifies  the  number  of  years  a  group  has  served  in  the  Air  Force,  and  is 
essentially  analogous  to  the  AFMPS  measure  of  Total  Active  Federal  Military  Service 
(TAFMS).  Tbiriy  year  groups  are  delineated  in  ISEM-P. 

Each  base  in  ISEM-P  represents  an  actual  Air  Force  base.  It  possesses  all  pertinent 
geographic,  organizational,  and  tour-length  characteristics  of  its  real  counterpart  The 
missions  assigned  to  the  simulated  bases  correspond  to  the  missions  attached  to  the 
analogous  real  bases,  and  the  groups  of  personnel  associated  with  the  simulated  bases 
conform  to  the  personnel  actually  assigned  to  the  applicable  mission  functions  at  the  real 
bases.  In  total,  17  bases  are  included  in  ISEM-P.  Of  these,  13  are  located  in  the  Continental 
United  States  (CONUS),  two  in  Europe,  and  two  in  the  Pacific.  Moreover,  of  the  13 
CONUS  bases,  three  are  affiliated  with  the  Air  Training  Command  (ATC),  three  with  the 
Military  Airlift  Command  (MAC),  three  with  the  Strategic  Air  Command  (SAC),  and  four 
with  the  Tactical  Air  Command  (TAC). 
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Time-On-Station  (TOS)  is  the  number  of  months  a  person  has  resided  at  a  base.  This 
property  is  used  by  the  AFMPS  to  determine  when  the  person  may  be,  or  must  be,  moved 
to  another  base.  Within  ISEM-P,  a  TOS  frequency  distribution  is  dynamically  maintained 
for  each  personnel  group  at  each  base. 

Thus,  the  structure  of  the  personnel  force  at  any  time  is  represented  in  ISEM-P  in 
terms  of  the  composition  of  the  stock  of  personnel  relative  to  the  preceding  five  properties: 
skill,  grade,  year  group,  base,  and  TOS.  These  are  the  basic  dimensions  within  which  the 
representations  of  Air  Force  decision  rules  contained  in  ISEM-P  operate.  The 
representations  of  decision  rules  included  in  ISEM-P  are  all  essentially  deterministic. 
Stochastic,  or  random,  elements  have  been  included  in  only  a  few  simulated  decision  rules 
as  default  conditions. 

4.4.4  ISEM-P  Operation 

Six  basic  types  of  decisions  routinely  made  by  the  AFMPS  are  represented  in  ISEM-P. 
They  are:  personnel  authorization  decisions,  airman  and  officer  procurement  decisions, 
promotion  decisions,  training  decisions,  transfer  decisions,  and  separation  decisions. 

The  first  five  types  of  decisions  are  organizational  decisions  expressly  taken  to  promote 
the  objectives  of  the  AFMPS.  Conversely,  the  last  type,  separation  decisions,  are  principally 
actions  taken  by  individual  Air  Force  personnel  to  serve  their  own  private  interests.  They 
are  not  directly  determined  by  the  AFMPS,  and  hence  may  or  may  not  be  compatible  with 
AFMPS  objectives.  Management  decisions  are  made  at  numerous  levels  in  the  AFMPS, 
including  the  Air  Staff,  major  command,  and  base  levels.  Each  level  operates  on  a  different 
time  scale,  considers  a  different  planning  horizon,  and  uses  a  different  aggregation  of  the 
force  structure  in  its  decision-making. 

Two  levels  of  decision-making  are  represented  in  ISEM-P.  They  have  been 
implemented  as  two  distinct  submodels:  the  aggregate  submodel  and  the  assignment 
submodel.  The  operation  of  these  submodels  is  explained  in  the  following  sections 

4.4.4. 1  The  Aggregate  Submodel  -  The  aggregate  submodel  operates  on  a  yearly  cycle. 
It  develops  and  actuates  long-range  force  structure  plans  for  a  horizon  period  corresponding 
to  the  end  of  the  current  simulated  year.  In  this  planning  process,  the  stock  of  personnel  is 
described  in  terms  of  an  array  called  the  inventory,  whose  three  dimensions  are  skill,  grade, 
and  year  group.  At  the  start  of  each  submodel  cycle,  the  inventory  describes  the  population 
present  at  the  beginning  of  a  simulated  year.  Each  cell  in  the  array  indicates  the  size  of  the 
group  having  the  skill,  grade,  and  year  group  properties  that  index  the  cell. 

Within  ISEM-P,  the  FYDP  is  represented  as  five  yearly  mission  plans  and  five  annual 
authorization  ceilings.  The  mission  plans  summarize  the  programs  included  in  the  FYDP  by 
stating  which  missions  are  to  be  attached  to  which  bases  during  each  simulated  year.  The 
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authorization  ceilings,  conversely,  express  the  budgetary  restraints  specified  in  the  FYDP  in 
terms  of  the  authorized  end-strengths  for  airmen  and  for  officers  in  each  simulated  year. 
Authorized  end-strength  is  the  maximum  total  number  of  personnel  allowed  to  be  in  the 
force  performing  mission  functions  at  the  end  of  a  year. 

Three  principal  AFMPS  policy  directives  are  embodied  in  the  submodel.  When  total 
manpower  requirements  exceed  authorization  ceilings,  manpower  polity  establishes  the 
priorities  attached  to  different  skills  in  determining  detailed  authorizations.  Next,  Air  Force 
promotion  policy  stipulates,  for  each  grade,  the  specific  year  groups  from  which  promotions 
into  the  next  higher  grade  may  be  drawn.  Finally,  the  equal  promotion  opportunity  policy 
requires  that  all  airmen  in  any  grade  are  eligible  for  promotion  on  the  same  terms  regardless 
of  the  skill  in  which  they  have  been  trained.  Representations  of  all  three  of  these  policies 
are  incorporated  into  the  simulated  decision  rules  contained  in  the  submodel. 

The  aggregate  submodel  operates  in  two  phases:  the  planning  phase  and  the  flow  phase. 
Thus,  for  each  simulated  year,  a  year  plan  is  formulated  based  on  the  initial  inventory  and 
the  mission  plan  for  that  year.  Then,  the  year  plan  is  executed  on  the  inventory  to  produce 
simulated  personnel  flows.  The  resultant  array  describes  the  inventory  at  the  end  of  the 
current  year,  which,  of  course,  corresponds  to  the  initial  inventory  for  the  next  submodel 
cycle. 

More  specifically,  each  simulated  year  the  aggregate  submodel  receives  as  inputs  the 
mission  plan  and  authorization  ceilings  for  that  year,  and  proceeds  to  perform  five  basic 
operations.  In  these  operations,  the  submodel  first  converts  the  mission  plan  into  manpower 
requirements  and,  then,  transforms  the  requirements  into  detailed  authorizations.  Next,  it 
projects  the  expected  state  of  the  inventory  after  anticipated  attrition.  Third,  it  formulates 
promotion,  training,  and  recruiting  plans  designed  to  change  the  state  of  the  inventory  to 
conform  to  the  detailed  authorizations.  It  then  removes  actual  separation  losses  from  the 
inventory.  Finally,  it  applies  the  formulated  plans  to  the  inventory.  This  process  produces 
the  inventory  that  serves  as  a  primary  input  to  the  next  annual  planning  cycle.  The 
formulated  plans  and  actual  results  are  also  passed  as  inputs  to  the  assignment  submodel. 

4.4 .4.2  The  Assignment  Submodel  -  The  assignment  submodel  deals  primarily  with  the 
geographic  distribution  of  personnel.  It  operates  on  a  monthly  cycle.  It  develops  and 
implements  short-range  plans  for  personnel  flow  among  bases  for  a  horizon  period  nominally 
nine  months  ahead  of  the  current  simulated  month.  In  this  submodel,  personnel  groups, 
identified  by  skill  and  grade,  are  distributed  among  the  bases  to  derive  the  stock  of  personnel 
at  each  base,  termed  the  base  supply. 

The  base  supply  is  described  a  two-dimensional  array.  Each  cell  in  the  array  indicates 
for  that  base  the  size  of  the  group  possessing  the  skill  and  grade  characteristics  that  index  the 
cell.  Each  personnel  group  within  each  base  supply  is  further  stratified  by  time-on-station, 
as  explained  in  Section  6.4.4. 1.  Personnel  in  training  are  accounted  for  in  the  enrollment  of 
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the  school  they  are  attending.  Schools  are  located  at  particular  bases,  but  their  enrollments 
are  not  included  in  the  base  supplies  of  those  bases. 

The  purpose  of  the  assignment  submodel  is  to  simulate  the  decision  procedures  that 
produce  the  assignment  orders  that  cause  personnel  to  be  transferred  between  bases.  In 
most  instances,  interbase  transfer  decisions  are  based  on  deviations  that  arise  between  base 
supplies  and  the  authorizations  at  the  bases.  Each  ISEM-P  base  has  an  authorization, 
determined  by  the  aggregate  submodel  during  its  planning  phase,  that  specifies  the  desired 
number  of  personnel  in  each  skill  and  grade  that  should  be  present  at  the  base  at  the  end 
of  each  simulated  year. 

In  addition  to  the  policy  directives  embodied  in  the  authorizations  and  plans  transmitted 
from  the  aggregate  submodel,  four  other  notable  AFMPS  policies  are  incorporated  in  the 
assignment  submodel.  Rotation  policy  establishes  restrictions  on  the  maximum  permissible 
length  of  time  personnel  may  serve  at  overseas  bases.  In  addition,  time-on-station  policy 
specifies  the  minimum  allowable  tour  length  for  personnel  residing  at  CONUS  bases.  Third, 
in  each  skill  and  grade  experiencing  an  overall  shortage  or  surplus  of  trained  personnel, 
world-wide  manning  level  policy  governs  the  sharing  of  that  surplus  or  shortage  among  bases. 
Finally,  interbase  transfer  policy  defines  the  order  in  which  different  categories  of  personnel 
are  selected  for  assignment  to  CONUS  and  overseas  bases.  The  simulated  decision  rules 
included  in  the  assignment  submodel  contain  representations  of  all  four  of  these  policy 
directives. 

Deviations  between  base  supplies  and  authorizations  can  occur  in  the  prototype  for 
several  reasons.  The  reasons  include:  separations,  assignments  of  cross-trainees  to  schools, 
changes  in  mission  plans  or  authorization  ceilings,  extraordinary  assignment  actions  such  as 
those  involved  in  closing  a  base,  and  restrictions  on  the  maximum  length  of  time  that 
personnel  may  serve  at  certain  bases. 

To  eliminate  deviations  of  base  supplies  from  authorizations  arising  from  any  of  these 
sources,  the  assignment  submodel  produces  appropriate  assignment  orders  and  then  executes 
the  orders  on  the  actual  pool  of  personnel.  Execution  of  an  assignment  order  effects  the  flow 
of  a  group  of  personnel  through  a  school  or  from  one  base  to  another. 

Like  the  aggregate  submodel,  the  assignment  submodel  operates  in  both  a  planning 
phase  and  a  flow  phase.  In  each  simulated  month,  the  planning  phase  creates  assignment 
orders  based  on  its  projection  of  base  supplies  for  the  horizon  period  nine  months  in  the 
future.  The  flow  phase  then  executes  the  assignment  orders  established  nine  months  earlier 
for  the  current  month. 

More  precisely,  the  assignment  submodel  converts  the  plans  formulated  for  each 
simulated  year  in  the  aggregate  submodel  into  month-by-month,  base-by-base  schedules  for 
separations,  promotions,  and  training  assignments.  It  also  schedules  all  transfers  among 
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bases  necessitated  by  policies  restricting  the  length  of  overseas  tours,  and  allocates  newly 
trained  personnel  to  bases.  The  schedules  are  established  for  the  horizon  period  on  the  basis 
of  projections  of  base  supplies  and  base  authorizations  for  that  period.  Then,  the  schedules 
are  applied  to  the  actual  base  supplies  to  simulate  the  resultant  flows  of  personnel  through 
training  and  among  bases. 

ISEM-P  assumes  that  all  airman  and  officer  procurement  demands  will  be  fulfilled,  in 
terms  of  both  quantities  and  aptitudes.  In  an  analogous  manner,  separations  in  ISEM-P  are 
completely  controlled  by  an  exogenously  supplied  set  of  separation  rates  for  each  group  in 
the  inventory. 

4.4.43  The  Consolidated  ISEM-P  System  -  The  basic  operation  of  ISEM-P,  focusing 
on  its  two  submodels  and  their  interrelationships,  is  displayed  schematically  in  Figure  6.1. 
This  figure  reveals  a  basic  difference  between  the  information-feedback  opportunities 
available  within  the  inventory  and  the  base  supply  planning  processes.  Because  the 
inventory  planning  horizon  corresponds  exactly  to  the  length  of  the  inventory  planning  cycle, 
each  inventory  plan  developed  in  the  aggregate  submodel  is  based  on  the  actual  state  of  the 
inventory  realized  at  the  end  of  the  preceding  cycle.  In  contrast,  the  base  supply  planning 
horizon  is  equal  in  length  to  nine  base  supply  planning  cycles.  Consequently,  each  base 
supply  plan  formulated  in  the  assignment  submodel  necessarily  depends  on  the  plans  and 
projections  developed  for  the  eight  intervening  months  between  the  current  simulated  month 
and  the  planning  horizon.  The  resultant  difference  in  information-feedback  mechanisms  is 
highlighted  in  Figure  6.1,  where  a  single  integrative  feedback  loop  appears  within  the 
aggregate  submodel,  while  two  essentially  independent  loops  are  depicted  for  the  base  supply 
planning  and  base  supply  flows  portions  of  the  assignment  submodel. 

There  are  12  monthly  cycles  of  the  assignment  submodel  for  each  yearly  cycle  of  the 
aggregate  submodel.  The  entire  process  is  iterated  through  the  total  number  of  simulated 
years,  typically  five,  for  which  input  data  are  provided  in  ISEM-P. 

Taken  together,  the  two  component  submodels  of  the  prototype  simulate  the  interaction 
of  many  of  the  goals,  objectives,  and  constraints  of  the  AFMPS.  However,  as  explained 
above,  each  submodel  addresses  a  distinct  subset  of  those  concerns;  and  even  where  both 
submodels  examine  the  same  issue,  the  manner  in  which  the  issue  is  resolved  may  vary. 

4.5  Training  Analysis  Support  Computer  System  (TASCS) 

The  Training  Analysis  Support  Computer  System  (TASCS)  is  a  microcomputer-based 
Instructional  System  Design  (ISD)  tool  (Logicon,  1985).  Its  purpose  is  to  support  detailed 
task  analysis  and  training  system  design.  It  is  intended  for  use  by  instructional  systems 
designers,  working  with  subject-matter  experts  (SMEs).  Task  analysis,  as  used  here,  refers 
to  the  process  whereby  job  tasks  are  translated  into  training  behaviors,  standards,  and 
conditions. 
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Aggregate  Submodel 


Figure  4.1  The  ISEM-P  Model 
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4.5.1  Relational  Data  Base  Management 


TASCS  is  primarily  a  task  analysis  data  base  management  system.  It  was  originally 
developed  by  Logicon,  Inc.  for  the  Air  Force  Ballistic  Missile  Office.  It  has  since  been 
refined  under  other  Air  Force  contracts.  The  version  described  by  Logicon  (1985)  was 
written  in  dBase  III,  a  powerful  general-purpose  microcomputer-based  relational  data  base 
tool. 

4.5.2  TASCS  Data  Content 

Logicon  (1985)  presented  a  number  of  reports  generated  by  TASCS  for  the  F16  task 
analysis.  One  such  report  is  a  training  objective  hierarchy.  Another  report  presents  the 
following  information  for  each  objective  behavior: 

1.  objective  training  sequence 

2.  conditions  under  which  behavior  is  performed 

3.  performance  measures  and  standard 

4.  proficiency  level 

5.  performance  frequency 

6.  performance  cues 

7.  who  performs 

8.  learning  subcategory  (list  of  standard  categories,  i.e.,  remember 

facts,  use  aided  procedures,  use  unaided  principles) 

9.  difficulty  rating  and  reasons  for  difficulty 

10.  safety  and  mission  criticality 

11.  related  systems  and  facilities 

12.  time  to  perform 

13.  job  code 

14.  team 

15.  source 

16.  remarks 

In  general,  these  data  are  entered  into  TASCS  by  instructional  designers  in  consultation 
with  SMEs. 

4.5.3  Relationships  with  IPS 

The  TDS  is  not  intended  to  service  all  ISD  processes.  Rather,  the  TDS  supports 
decisions  made  early  in  the  training  development  process  (Step  1  and  part  of  Step  2)  that 
result  in  constraints  within  which  the  full  ISD  process  must  be  done.  As  such,  the  TDS 
would  be  used  before  TASCS;  TASCS  would  be  used  to  support  curriculum  development 
after  certain  decisions  are  made  using  the  TDS. 
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Some  of  the  data  entered  into  TASCS  can  be  obtained  from  the  TDS  or  the 
occupational  survey  data  base.  Examples  of  these  data  include  proficiency  level,  difficulty, 
frequency,  and  criticality.  If  TASCS  were  to  be  used  in  conjunction  with  TDS,  as  suggested 
here,  it  would  be  advantageous  for  certain  TDS  data  to  be  input  directly  and  automatically 
into  TASCS. 


4.6  Other  Air  Force  Modeling  R&D 

The  Air  Force  manpower,  personnel,  and  training  (MPT)  community  could  make  use 
of  the  TDS  technology  in  many  different  areas.  One  area  of  primary  interest  to  AFHRL 
would  be  the  potential  use  of  TDS  in  support  of  Air  Force  MPT  modeling.  Several  potential 
applications  are  described  below. 

4.6.1  Time  to  Proficiency  (XEE)  Model 

Under  contract  with  several  contractors,  AFHRL  has  developed  a  modeling  tool  that 
could  be  useful  to  personnel  and  manpower  planners  in  selecting  optimum  qualification 
scores  for  classification  into  an  Air  Force  Specialty  (AFS).  This  model  tool  was  developed 
originally  by  Monaco  and  Carpenter  (reported  in  Monaco,  et  al.,  1989)  as  a  demonstration 
of  using  a  learning  curve  to  measure  proficiency  and  serving  as  a  predictor  of  productive 
capacity.  The  original  model  has  been  improved  and  extended  by  Stone  and  his  coworkers 
(Stone,  et  al.  1989)  to  include  optimization  over  multiple  AFS’s  with  a  computerized 
prototype.  This  tool  shows  remarkable  promise  for  the  ability  of  manpower  and  personnel 
planners  to  use  productive  capacity  and  cost  as  good  measures  to  set  minimum  qualifications 
for  AFS  classifications. 

The  costs  used  in  the  development  of  this  model,  however,  suffer  from  lack  of 
differentiability  across  AFS’s  and  show  little  effect  of  differential  training  patterns.  An  ideal 
match  for  the  enhanced  usefulness  of  the  TTP  model  would  be  a  combination  of  the  TDS 
and  TTP  into  a  single  policy  tool  that  could  be  used  to  set  minimum  qualifications  for 
classification  standards  and  also  explore  the  effect  of  alternate  training  patterns  on  the 
classification  standards.  Although  this  combination  would  take  a  great  deal  of  programming 
effort,  an  earlier  version  could  be  developed  that  summarized  several  training  patterns  for 
the  AFS’s  currently  used  in  the  TTP  model,  and  that  information  about  costs  could  be  used 
to  enhance  the  existing  TTP  model. 

4.6.2  Person-Job  Match  Modeling 

The  Air  Force  has  been  using  decision  modeling  techniques  to  determine  optimal  job 
classifications  of  recruits  for  some  period  of  time.  The  earliest  reported  version  of  an 
optimal  batch  assignment  model  had  been  developed  and  reported  on  in  Ward  (1959)  and 
Bottenberg  (1962).  This  process  used  a  linear  programming  methodology  to  allow 
classification  clerks  at  Lackland  AFB  to  assign  graduates  of  Basic  Military  Training  (BMT) 
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to  formal  training  slots  in  some  optimal  fashion,  based  on  availability  of  slots  and  time  of 
graduation.  Another  attempt  at  optimal  classification  involved  a  formal  computer  model,  the 
Training  Line  Simulator,  developed  by  Hatch,  et  al.  (1974)  using  modem  linear  programming 
methodologies  to  perform  this  same  process.  The  training  line  simulator  was  later  replaced 
with  a  non-optimal  assignment  process  called  the  PACE  system,  which  is  still  in  effect  today 
for  all  classification  decisions  made  during  BMT. 

The  decision  index  developed  by  Ward  (1959)  was  later  used  by  Ward  and  Haltman 
(1974)  to  form  what  was  to  become  the  Procurement  Management  Information  System 
(PROMIS).  In  the  PROMIS  software  package,  the  Air  Force  was  able  to  combine  decision 
modeling  with  optimization  techniques  to  allow  for  a  sequential  classification  algorithm  to 
be  used  prior  to  enlistment.  The  fully  developed  PROMIS  system  (described  in  Ward,  et  al., 
1979)  is  basically  the  system  still  in  use  today  by  the  Air  Force  Recruiting  Service  to  perform 
the  initial  classification  decision  for  approximately  50%  of  the  enlistees  into  the  Air  Force. 
The  other  50%  of  the  classification  decisions  are  still  made  in  non-optimal  form  using  PACE. 
Some  work  which  is  as  yet  unreported  is  currently  going  on  at  AFHRL  to  define  a 
post-enlistment  classification  process  which  would  replace  the  PACE  system  with  a  batch 
assignment  algorithm  that  used  some  of  the  optimization  features  of  the  PROMIS,  but  the 
cost  of  training  will  not  be  included  in  this  optimization  process. 

With  the  development  of  TDS,  the  Air  Force  has  an  opportunity  to  take  advantage  of 
a  significant  advance  in  the  state  of  the  art  in  classification  modeling.  The  TDS  software  in 
its  current  configuration  could  be  readily  modified  to  be  used  to  assist  in  the  post-enlistment 
person-job  match  process.  The  decision  to  classify  and  train  an  enlistee  today  is  made 
without  regard  to  the  cost  or  effectiveness  of  the  subsequent  training,  nor  does  the 
classification  process  allow  for  the  consideration  of  different  training  patterns.  By  modifying 
TDS  to  be  used  in  the  post-enlistment  process,  the  Air  Force  could  make  optimal 
classification  and  assignment  decisions  in  the  same  process,  and  have  the  optimal  process 
include  differential  training  cost  considerations  in  addition  to  the  factors  that  are  currently 
being  considered  by  AFHRL. 


4.7  MPT  Modeling  Integration 

As  noted  above,  the  TDS  might  be  used  or  adapted  for  a  variety  of  purposes  involving 
the  modeling  of  Air  Force  specialties.  If  implemented  as  an  extension  of  the  normal 
occupational  survey  process,  the  TDS  modeling  would  naturally  support  the  U&T  Workshop 
process  (as  now  authorized  by  the  new  AFR  50-23,  Enlisted  Specialty  Training,  August  1990). 
By  using  the  TDS  and  such  AFS  modeling  to  support  its  decision  making,  a  U&T  Workshop 
could  not  only  formulate  but  also  evaluate  a  variety  of  possible  job  and  training 
configurations  so  as  to  optimize  training  (resident  training,  FTDs,  OJT,  etc.)  in  ways  which 
would  best  meet  the  needs  of  the  specialty. 
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By  selecting  or  specifying  the  jobs  and  training  programs  of  an  AFS,  tne  U&T  Workshop 
would  be  performing  its  key  management  function,  but  would  also  be  completing  Step  1  of 
the  ISD  process.  It  would  also  be  selecting  (or  refining)  the  appropriate  Master  Task  List 
and  Generic  Position  Task  Lists  needed  as  the  initial  phase  of  AOTS  implementation  for  a 
career  field.  It  would  also  be  completing  the  initial  phase  of  development  of  a  Career  Field 
Training  Management  Plan  as  recommended  by  the  new  AFR  50-23).  Thus,  implementation 
of  the  TDS  could  and  should  lead  to  the  integration  of  several  different  R&D  programs  into 
an  integrated  operational  program. 

These  various  R&D  programs  are  now  being  undertaken  as  separate,  isolated  projects. 
A  more  realistic  approach  would  be  to  design  a  general  integration  of  all  MPT  research 
initiatives  which  would  aim  at  the  long-term  synthesis  of  the  various  systems  into  a  coherent 
operational  system.  Such  a  strategy  would  also  facilitate  the  identification  of  other  areas 
where  present  R&D  efforts  do  not  yet  provide  needed  advanced  technology  required  for  such 
an  overall  operational  implementation.  The  TDS  might  well  serve  as  the  basis  for  such  an 
overall  plan  and  integration. 

In  addition,  we  need  to  examine  the  potential  uses  of  TDS  outside  of  its  design 
environment;  that  is,  in  other  potential  applications  beyond  support  of  technical  training 
decision  making.  One  obvious  possible  application  would  be  its  use  in  the  New  Weapon 
System  Acquisition  process,  and  several  requests  have  already  been  made  by  members  of  that 
community  for  development  of  such  an  advanced  TDS  capability.  We  should  also  explore 
other  uses  for  TDS  and  training  decisions  modeling  technologies  within  the  Air  Force  (officer 
specialties,  professional  military  education,  etc.)  as  well  as  potential  applications  in  other 
services,  other  DOD  agencies,  and  transfer  of  this  technology  to  the  civilian  world  as  well. 
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5.0  EXERCISING  THE  TDS 


A  major  testing  of  the  technology  and  the  software  involved  in  the  proof-of-concept  TDS 
was  undertaken  to  identify  any  problems  in  the  system  itself  or  in  its  software.  The 
objectives  of  this  test  were  satisfied,  some  software  problems  were  identified  and  corrected, 
and  several  areas  of  potential  improvement  were  identified.  This  test  of  the  system  is 
discussed  in  detail  below. 

The  Training  Decisions  Technology  Analysis  Research  Plan  specified  that  TDS  software 
resident  on  the  AFHRL  UNISYS  computer  would  be  executed  for  the  current  U&T  pattern 
and  selected  other  patterns  for  the  four  TDS  specialties,  as  a  basis  for  displaying  system 
capabilities,  developing  examples  of  products,  and  identifying  any  software  problems.  There 
were  four  major  subtaskings  involved  in  this  major  exercise  of  the  TDS: 

(a)  run  all  TDS  software  programs,  as  well  as  selected  options  for 
optimization  problems; 

(b)  enhance  and  refine  the  current  version  of  the  TDS  software,  identifying 
and  correcting  any  software  problems; 

(c)  document  TDS  analyses  and  optimization  runs  for  the  current  and 
several  alternative  U&T  patterns  for  each  of  the  four  TDS  specialties;  and 

(d)  develop  a  discussion  of  other  impacts  upon  U&T  specialties  including 
the  impact  of  constrained  TDY-to-school  resources  upon  relevant  cost  and 
capacity  variables. 


5.1  Approach  and  Results 

The  basis  of  this  initial  exercise  was  to  employ  procedures  specified  in  the  August  1988 
Procedural  Guide.  TDS  User  Instructions  (Task  IV,  CDRL  1:25),  in  order  to  check  the 
validity  of  the  instructions  and  to  help  identify  any  errors  or  omissions.  As  stated  in  the 
Research  Plan,  each  specialty  was  to  be  tested  using  the  current  utilization  and  training 
pattern,  the  two  or  three  alternate  utilization  and  training  patterns  most  preferred  by  field 
managers  (as  indicated  in  the  earlier  U&T  Pattern  Preference  Survey  administered  in  the 
original  TDS  R&D  effort),  and  a  U&T  pattern  that  depicted  a  25  percent  reduction  of 
personnel  TDY  to  Field  Training  Detachments  and  other  advanced  courses  (TDY-to-School 
funds).  In  addition,  an  alternative  with  25  percent  bypassing  the  ABR  and  receiving  most 
of  their  training  via  OJT  was  examined.  Therefore,  each  specialty  actually  involved  modeling 
five  or  six  different  U&T  patterns. 

Results  from  the  initial  runs  were  then  compared  with  the  basic  runs  from  the  initial  TDS 
R&D  to  assure  that  the  system  was  operating  properly.  Several  of  these  tests  are  described 
below. 
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5.1.1  AFS  423X1.  Environmental  Systems  Maintenance  Exercise 


The  capabilities  of  the  TDS  programs  were  exercised  with  AFS  423X1,  Environmental 
Systems  Maintenance.  All  AFS  423X1  data  files  resided  on  the  AFHRL  UNISYS 
equipment,  with  the  exception  of  the  TDY-to-school  reduction  and  the  additional  model  for 
reducing  ABR  attendance  by  25  percent.  Two  new  data  files  were  created  to  model  the 
TDY-to-school  reduction  and  ABR  reduction. 

The  approach  taken  for  each  U&T  pattern  was  to  put  together  one  runstream  for  all 
subsystems  of  the  TDS  and  have  the  FUS,  TCS,  and  RCS  outputs  accomplished  at  one  time. 
The  UTPSIM  program  comprised  the  first  part  of  the  runstream,  the  TRNPRF  program  the 
second  segment,  and  the  RCS  programs  a  third  part.  By  exercising  all  programs  at  one  time, 
we  could  compare  errors  or  inconsistencies  with  the  draft  user’s  guide  and  identify 
corrections  needed. 

[Note:  An  example  runstream  used  to  produce  U&T  pattern  reports  and  a  complete  set 
of  reports  for  AFS  423X1  have  been  published  in  the  TDS  Examples  report,  a 
companion  document  for  the  TDS  User  Manual  and  the  TDS  Programmer  Guide  to 
provide  examples  of  each  type  of  TDS  report.  These  examples  are  available  from 
AFHRL/IDET,  Brooks  AFB  TX  78235-5601.] 

5.1. 1.1  AFS  423X1  Exercise  Problems  -  Some  difficulties  were  encountered  in  the  initial 
efforts  to  exercise  the  system  and  produce  the  AFS  423X1  TDS  reports. 

There  were  some  unanticipated  errors  in  example  runstreams  included  in  the  August  1988 
Procedural  Guide.  TDS  User  Instructions  (Task  IV,  CDRL  1:25),  which  terminated  the  initial 
runs.  Some  of  these  were  typographical  errors  made  in  preparing  the  Guide,  which  could 
be  easily  corrected.  Others  involved  some  disparity  in  the  names  and  descriptions  of  two 
required  data  matrices;  these  errors  were  identified  and  corrected;  they  have  also  been 
corrected  in  a  revision  of  the  procedural  guide  (the  TDS  Programmer  Guide.  CDRL  A007). 

The  RCS  capacity  programs,  which  had  been  improved  on  the  CONSAD  and  MDC  VAX 
machines  since  the  final  TDS  Progress  Review  (Sept  1988),  had  not  been  fully  upgraded  on 
the  AFHRL  UNISYS  equipment.  Since  there  were  still  several  known  problems  with  this 
software,  we  decided  these  problems  needed  to  be  corrected  before  transfer  of  the  upgraded 
software  to  the  AFHRL  equipment.  Correction  of  these  programs  proved  more  difficult  than 
anticipated  and  took  more  time  than  expected.  The  corrected  programs  were  completed  by 
5  May  1989  and  were  uploaded  to  the  UNISYS  the  following  week.  Such  uploading  includes 
recompiling  the  software  using  AFHRL-unique  language;  some  difficulty  was  encountered 
in  compiling  one  of  the  capacity  programs  which  further  delayed  the  process.  As  of  12  May 
1989,  all  of  the  capacity  programs  had  been  successfully  compiled  and  are  now  operational 
on  the  AFHRL  UNISYS  machine. 

Some  problems  were  encountered  in  identifying  the  appropriate  files  stored  in  the  AFHRL 
UNISYS;  during  the  original  TDS  R&D  and  the  TDS  training  session,  many  runs  were  made 
which  were  still  on  the  system.  These  many  different  runs  were  labelled  in  different  ways, 
depending  on  which  analyst  had  ordered  the  run  and  when  the  run  was  made  (many  of  the 
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runs  made  during  TDS  training  or  in  preparation  for  the  final  TDS  progress  review  had 
unique  or  abbreviated  labels  since  they  were  temporary  files).  The  amount  of  time  since  the 
files  were  last  used  was  a  factor,  since  some  of  the  temporary  file  names  had  not  been 
recorded  or  were  not  remembered  adequately.  The  master  listing  of  current  files  provided 
by  AFHRL/IDET  was  very  useful  in  resolving  most  of  this  kind  of  problem,  but  considerable 
research  and  testing  was  required  to  assure  that  the  proper  file  had  been  identified  and  used 
in  making  runs.  The  number  of  files  involved  in  multiple  U&T  patterns  for  each  AFS  makes 
the  file  inventory  complex;  we  highly  recommend  that  a  systematic  approach  be  used  in 
naming  TDS  files  [analogous  to  the  CODAP  conventions  for  file  designations]. 

A  related  issue  is  the  labelling  of  TDS  products;  the  current  software  was  set  up  in  such 
a  way  that  all  U&T  pattern  products  included  a  subtitle  for  "Current  U&T  Pattern"  which, 
in  this  case,  meant  the  pattern  being  analyzed  in  that  particular  run  -  the  one  being  studied 
at  that  point.  This  became  a  problem  when  multiple  runs  were  compared;  to  keep  the 
various  alternatives  in  mind,  the  analyst  must  refer  to  the  initial  page  of  the  product  and 
decipher  the  filename  and  then  mark  each  product.  A  better  approach  would  be  to  insert 
a  unique  title  line  for  each  run  for  a  given  U&T  pattern.  This  would  prevent  any  confusion 
in  the  identity  of  the  AFS  or  U&T  pattern  being  studied  by  having  a  clear  text  designator 
for  the  AFS  and  U&T  pattern  on  the  product. 

5. 1.1.2  AFS  423X1.  Environmental  Systems  Maintenance  Results  -  For  the 
Environmental  Systems  Maintenance  specialty,  the  following  AFS  models  were  developed 
and  run: 

a.  Current  U&T  Pattern 

b.  Alternative  1  U&T  Pattern  (Two  track  -  Tactical  or  Strategic/ Airlift) 

c.  Alternative  2  U&T  Pattern  (Increased  OJT/FTD  Role) 

d.  Alternative  4  U&T  Pattern  (No  Technical  School) 

e.  Decreased  TDY-to-school  (modeled  as  25%  reduced  flow  into  FTDs,  PME, 

and  advanced  courses) 

f.  Decreased  ABR  Flow  (25%  direct  to  duty  bypassing  the  ABR). 

A  review  of  the  Current  U&T  Pattern  and  Alternative  U&T  Patterns  1,  2,  and  4, 
indicated  that  results  were  comparable  to  data  output  in  the  basic  TDS  R&D  (with  minor 
variance  where  some  known  change  had  been  made  or  error  corrected).  These  data  are 
available  for  review  in  the  data  runs  enclosed  with  this  report. 

The  two  new  AFS  models  (e.  and  f.  above)  are  of  particular  interest,  since  they 
represent  new  tests  of  the  TDS  software  with  specified  restrictions  or  changes  to  the 
current  U&T  pattern.  To  implement  such  changes,  it  was  necessary  to  modify  the  data 
files  for  the  specialties. 

For  the  decreased  TDY-to-school  alternative,  we  elected  to  model  this  change  as  a 
decrease  of  .25  in  the  probability  of  attending  an  FTD  course  (there  are  no  advanced 
courses  in  the  AFS  423X1  TDS  current  U&T  model).  The  file  reflecting  school 
attendance  probabilities  was  located  (in  this  case,  a  job-driven  training  file),  and  each 
probability  reduced  accordingly.  PME  attendance  is  reflected  in  a  second  file 
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(TAFMS-driven  training  file);  these  probabilities  were  also  reduced  by  .25.  Theoretically, 
these  reduction  in  the  proportion  of  airmen  in  a  job  attending  an  FTD  and  airmen  in  a 
TAFMS  group  attending  a  PME  course,  should  equate  to  an  overall  reduction  of  25 
percent  in  TDY-to-school  travel  for  this  specialty.  In  practice,  there  will  of  course  be 
some  variation  in  results  (plus  or  minus  a  percent  or  so)  due  to  rounding  or  other 
computational  factors. 

For  the  reduced  ABR  attendance  model,  only  one  data  matrix  had  to  be  modified;  this 
is  the  entry  file  which  reflects  the  average  number  of  persons  who  enter  the  career  field 
each  month  and  how  they  are  allocated  (to  ABR  or  to  AFS  jobs).  Normally,  for  AFS 
423X1,  almost  all  (96%+)  enter  the  career  field  by  attending  the  ABR  course;  only  a 
small  number  (normally  individuals  with  prior  service  or  experience)  bypass  the  ABR  and 
are  given  direct  duty  assignments  (DDAs).  Since  the  current  U&T  model  already 
provided  for  some  DDAs,  the  only  changes  needed  to  model  this  alternative  were  to 
decrease  the  probability  of  attending  the  ABR  by  .25  and  to  reapportion  the  flow  of 
individuals  directly  into  field  jobs  [a  decision  had  to  be  made  as  to  how  to  reapportion; 
we  chose  to  use  the  same  proportion  for  each  job  as  in  the  current  flow  of  DDAs.  An 
alternative  strategy  might  be  to  use  the  relative  proportion  of  1st  job  personnel  in  each 
job  times  .25  to  estimate  the  new  DDA  flow  for  each  job.] 

The  results  appear  reasonable  and  understandable;  the  approach  taken  to  modeling 
the  change  in  the  specialty  appears  to  work  reasonably  well.  Two  tables  comparing  the 
current  U&T  pattern  (as  a  baseline)  to  the  alternatives  are  shown  below  to  highlight  the 
results  (see  Tables  5.1  and  5.2).  The  Formal  School  Costs  portion  of  these  tables  was 
extracted  from  the  RCS  Cost  Report  for  both  the  current  and  new  U&T  model;  the  OJT 
Costs  were  extracted  from  the  second  section  of  the  same  report.  The  totals  and 
difference  ($  and  %)  values  were  used  to  compile  current  and  new  U&T  model  reports 
to  provide  a  summary  comparison  between  these  two  alternatives. 


25%  REDUCTION  IN  TDY-TO-SCHOOL  COSTS  * 
AFS  423X1,  Environmental  Systems  Maintenance 

(in  Dollars) 


Formal  School  Costs 

OJT  Costs 

Total  AFS 

Trainee 

Trainer 

Per  Diem 

Transport 

Trainee 

Trainer 

Drag  Costs 

DTC** 

DTC 

Costs 

Costs 

OJT  Cost 

OJT  Cost 

Per  Year 

CURRENT  U&T 

962400 

119,031 

662435 

387411 

334470 

221453 

52446400 

REDUCED  TRAVEL 

851,504 

100,236 

572482 

292458 

338,720 

223401 

2478401 

Difference 

-  110/196 

-  18,795 

.  50,553 

-  94453 

+  4450 

+  2448 

-268499 

Percent  Dif. 

-  11.49% 

- 15.79% 

-  8.12% 

-  2453% 

+  147% 

+  L96% 

-  10.13% 

*  Modeled  as  a  25%  reduction  in  the  probability  of  attending  an  FTD,  PME,  or  advanced  course 
**  DTC  «  Direct  Training  Cost 

Table  5.1  25%  Reduction  in  TDY-to-School  Costs 
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For  AFS  423X1,  a  reduction  in  TDY-to-school  funding  would  slightly  increase  total  OJT 
costs  for  the  specialty  (about  1  percent)  and  would  result  in  a  decrease  of  about  10  percent 
in  total  AFS  training  costs  per  year.  This  reduced  TDY-to-school  model  indicates  that  most 
of  this  savings  is  the  result  of  reduced  trainee  and  trainer  time  in  the  classroom;  non-salary 
savings  would  be  $94,953  in  reduced  transportation  costs  (24.5%)  and  an  associated  $50,553 
in  per  diem  (8.1%). 

For  the  other  AFS  model  (25  percent  reduction  in  Basic  Resident  Course  attendance), 
comparable  data  are  shown  in  Table  5.2.  There  is  a  substantial  savings  in  formal  school 
costs,  with  the  greatest  savings  being  in  per  diem  costs  and  trainee  time.  Note  that  in  this 
specialty,  the  ABR  is  less  than  12  weeks  long  and  thus  is  considered  an  enroute  TDY  for 
students;  thus,  per  diem  is  handled  in  a  TDY-to-school  account  rather  than  normal 
operations  and  manpower  (O&M)  funding. 

[Other  AFSs,  such  as  AFS  328X4,  with  longer  than  12  week  ABR  courses,  are  handled 
as  a  permanent  change  of  station  (PCS);  in  such  cases,  travel  and  per  diem  are  managed  as 
O&M  and  are  not  included  in  TDY-to-school  funding.  Thus,  the  17  percent  savings  in  per 
diem  costs  is  something  of  an  artifact  of  the  cost  accounting  system  and  the  categorization 
of  schools  as  either  PCS  or  TDY  enroute.  For  an  AFS  like  328X4  with  a  PCS  school,  such 
savings  would  not  occur,  or  would  not  be  visible.] 

For  the  reduced  ABR  model,  there  is  a  substantial  increase  in  OJT  costs,  as  might  be 
expected  when  25  percent  bypass  the  technical  school. 

25%  REDUCTION  IN  BASIC  RESIDENT  SCHOOL  FLOW  * 

AFS  423X1,  Enviro— mtal  Systems  Mainteaawz 

(in  Dollars) 

Formal  School  Costs  OJT  Costs  Total  AFS 


Trainee 

DTC** 

Trainer 

DTC 

Per  Diem 
Costs 

Transport 

Costs 

Trainee 

OJT  Cost 

Trainer 

OJT  Cost 

Trng  Costs 
Per  Year 

CURRENT  U&T 

962,000 

119,031 

662435 

387411 

334470 

221453 

$2446400 

REDUCED  ABR 

831,241 

107,538 

516,482 

378465 

376471 

236,976 

2447473 

Difference 

-  130,759 

-  1M» 

-106453 

•  8446 

+  42401 

+  15,723 

•  198427 

Percent  Dif. 

-  13.59% 

■  9.66% 

-  173)5% 

•  248% 

+  12.71% 

+  7.11% 

-  741% 

*  Modeled  as  a  25%  direct  doty  assignments  (bypassing  the  ABR) 
**  DTC  »  Direct  Training  Cost 


Table  52  25%  Reduction  in  ABR  Attendance 


Note,  however,  that  the  increase  in  OJT  trainee  and  trainer  costs  is  only  slightly  less  than 
the  reduction  in  Formal  School  trainee  and  trainer  costs,  in  terms  of  percentages.  In  terms 
of  actual  dollar  values,  the  increase  in  OJT  indicates  much  less  of  an  increase  in  trainee 
time,  but  a  greater  gain  than  loss  (over  formal  school)  in  trainer  time.  Trainees  take  less 
hours  to  learn  the  tasks  involved  on  the  job,  but  more  trainer  time  and  effort  is  required. 
The  question  becomes,  then,  one  of  whether  the  field  units  have  the  capacity  to  conduct  the 
increased  OJT  required  under  this  scenario.  A  training  capacity  analysis  was  conducted  for 
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AFS  423X1  with  results  suggesting  that  there  would  not  be  any  significant  impact  on  the 
ability  of  the  representative  field  units  to  conduct  the  increased  OJT  required  by  such  a 
change.  Since  these  results  are  not  particularly  significant,  they  are  not  included  here;  for 
illustrative  purposes,  the  capacity  analyses  for  AFS  328X4  are  more  worthwhile  and 
interesting. 

5.1.2  AFS  328X4.  Avionic  Inertial  and  Radar  Navigation  Systems  Results 

For  the  Avionics  Inertial  and  Radar  Navigation  Systems  specialty,  the  following  AFS 
models  were  developed  and  run: 

a.  Current  U&T  Pattern 

b.  Alternative  1  U&T  Pattern  (Two  track  -  Tactical  or  Strategic/ Airlift) 

c.  Alternative  3  U&T  Pattern  (Versatile  Workforce;  Initial  assignment  to  TAC 

with  subsequent  assignments  to  SAC/MAC) 

d.  Alternative  5  U&T  Pattern  (Specialization  by  system) 

e.  Reduced  TDY-to-school  U&T  (modeled  as  .25  reduction  in  the  probability 

of  attending  an  FID,  PME,  or  advanced  course) 

f.  Reduced  ABR  Flow  U&T  (25%  bypass  ABR  for  DDAs) 

5.1.2.1  AFS  328X4  Training  Cost  Results  -  A  review  of  the  Current  U&T  Pattern  and 
Alternative  U&T  Patterns  1, 3,  and  5,  indicated  that  results  were  comparable  to  data  output 
in  the  basic  TDS  R&D  (with  minor  variance  where  some  known  change  had  been  made  or 
error  corrected).  As  with  AFS  423X1,  the  two  new  AFS  models  (e.  and  f.  above)  are  of 
particular  interest,  since  they  represent  new  tests  of  the  TDS  software  with  specified 
restrictions  or  changes  to  the  current  U&T  pattern.  To  implement  such  changes,  it  was 
necessary  to  modify  the  data  files  for  the  specialties  (as  was  done  with  AFS  423X1). 

The  results  appear  reasonable  and  understandable;  the  approach  taken  to  modeling  the 
change  in  the  specialty  appears  to  work  reasonably  well.  Two  tables  comparing  the  current 
U&T  pattern  (as  a  baseline)  to  the  alternatives  are  shown  below  to  highlight  the  results  (see 
Tables  5.3  and  5.4).  The  Formal  School  Costs  portion  of  these  tables  was  extracted  from 
the  RCS  Cost  Report  for  both  the  current  and  new  U&T  model;  the  OJT  Costs  were 
extracted  from  the  second  section  of  the  same  report.  The  totals  and  difference  ($  and  %) 
values  were  calculated  by  hand  from  the  compiled  current  and  new  U&T  model  reports  to 
provide  a  summary  comparison  between  these  two  alternatives. 

For  AFS  328X4,  the  reduced  TDY-to-school  U&T  option  indicates  a  25  percent  savings 
of  both  transportation  and  per  diem  costs  (see  Table  5.3).  In  this  case,  since  the  ABR  is 
a  PCS  course,  there  are  no  per  diem  expenses  associated  with  the  ABR;  thus,  the  observed 
value  for  per  diem  for  both  the  Current  U&T  and  the  Reduced  Travel  U&T  both  involve 
only  FTD,  PME,  and  advanced  courses.  Note  also  that  while  the  increase  in  OJT  Trainee 
dollar  costs  is  somewhat  less  than  the  drop  in  Formal  School  Trainee  costs,  the  opposite  is 
true  for  Trainer  costs  (OJT  vs  Formal  School).  The  overall  impact  on  total  AFS  training 
costs  per  year  in  terms  of  percentage  savings  is  relatively  small  (- 1.43%);  however,  we  also 
need  to  be  able  to  assess  the  impact  of  the  added  OJT  responsibility  for  field  supervisors 
(trainers).  This  involves  an  OJT  capacity  analysis,  which  will  be  reported  later. 
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25%  REDUCTION  IN  TDY-TO-SCHOOL  COSTS  * 
AFS  328X4,  Avionic  Inertial  &  Radar  Navigation  Systems 

(in  Dollars) 


Trainee 

Formal  School  Costs 
Trainer  Per  Diem 

Transport 

O.TT  Costs 
Trainee 

Trainer 

Total  AFS 
Trng  Costs 

DTC  *» 

DTC 

Costs 

Costs 

OJT  Cost 

OJT  Cost 

Per  Year 

CURRENT  U&T 

1,545*437 

159,318 

12414 

326487 

3*441*468 

1,003,679 

$6*489403 

REDUCED  TDY 

1*446,011 

146^03 

9,183 

242,796 

3,507,167 

1,024,022 

6496JI82 

Difference 

-  79*426 

-  12*415 

.3*431 

-83,791 

+  65499 

+  20,343 

•  93421 

Percent  Difference 

-  5.14% 

-  7.79% 

-27.19% 

-  2565% 

+  L91% 

-1-  2.03% 

-  143% 

*  Modeled  as  a  25%  reduction  in  the  probability  of  attending  an  FID,  PME,  or  advanced  coarse 
**  DTC  =  Direct  Training  Cost 


LN20COST328C.  12-17-90  vs  LN20COST328D.12-17-90 


Table  5.3  AFS  328X4  Reduced  TDY-to-school 


For  the  25  percent  reduction  in  ABR  flow  (Table  5.4),  note  that  this  change  has  no  real 
impact  on  the  total  AFS  training  costs  per  year  (plus  thirty-five  hundredths  of  one  percent 
additional  cost).  The  greatest  percent  difference  is  a  savings  of  about  19  percent  in  Formal 
School  trainee  hours;  however,  this  is  mostly  offset  by  the  7.69  percent  increase  in  OJT 
trainee  hours.  The  most  significant  increase  is  the  additional  $81,491  of  OJT  trainer  time 
required;  this  is  over  $54,000  more  than  the  savings  in  Formal  School  trainer  costs.  It  would 
appear  that  we  are  trading  savings  in  Trainee  time  (salary)  for  the  time  of  Trainers  in  the 
field;  thus  the  availability  of  OJT  trainers  in  operational  units  becomes  the  critical  factor. 
This  possible  constraint  needs  to  be  assessed  through  a  TDS  capacity  analysis. 


25%  REDUCTION  IN  BASIC  RESIDENT  SCHOOL  FLOW  • 
AFS  328X4,  Avionic  Inertial  &  Radar  Navigation  Systems 

(in  Dollars) 


Formal  School  Costs 

OJT  Costs 

Total  AFS 

Trainee 

Trainer 

Per  Diem 

Transport 

Trainee 

Trainer 

Trng  Costs 

PTC** 

DTC 

Costs 

Costs 

OJT  Cost 

OJT  Cost 

Per  Year 

CURRENT  U&T 

1*545*437 

159418 

12414 

326487 

3441*468 

1403479 

$6489403 

REDUCED  ABR 

1452444 

132*646 

14492 

3204M 

3,706466 

1485470 

6411406 

Difference 

-292,793 

•26472 

+  1478 

-  5499 

+  264498 

+  81491 

+  22403 

Percent  Difference 

•  18.94% 

-16.74% 

-  1.17% 

•  1.74% 

+  7.69% 

+  8.12% 

+  035% 

*  Modeled  as  a  25%  direct  doty  assignment  (bypassing  the  ABR) 

**  DTC  *  Direct  Training  Cost 


LN20COST32S.12-17-90  vs  LN20COST328T.12-19-90 


Table  5.4  25%  Reduction  in  ABR  328X4  Student  Flow 
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5. 1.2.1  Training  Capacity  Results  -  The  Training  Capacity  Component  of  the  RCS 
evaluates  the  capacities  of  various  representative  sites  to  provide  training  in  appropriate 
settings  on  different  combinations  of  TMs  and  in  training  volumes  that  are  compatible  with 
the  U&T  patterns  identified  in  the  FUS.  Inputs  to  this  component  consist  of  the  following: 
TM  combinations  and  training  volumes  for  the  various  U&T  patterns  (from  the  FUS), 
predicted  amounts  of  specific  resources  required  for  the  provision  of  training  on  each  TM 
in  each  training  setting  (in  the  form  of  regression  equations  from  the  Resource 
Requirements  Component),  and  availabilities  of  those  resources  for  providing  training  at 
each  representative  site.  Resource  availability  data  are  collected  in  a  Training  Resources 
Availability  survey  of  TTCs,  FTDs,  and  representative  field  units.  For  dedicated  training 
resources,  data  on  the  sharing  of  equipment  and  other  resources  must  be  collected  since 
sharing  has  a  potential  impact  on  training  capability,  and  may  vary  by  site. 

An  analyst  develops  estimates  of  the  capacity  of  each  representative  site  to  accommodate 
various  combinations  of  TMs  and  training  loads,  and  identifies  any  resource  limitations  that 
constrain  representative  sites  from  accommodating  particular  U&T  patterns.  When  such 
constraints  are  encountered,  they  are  displayed  in  the  OJT  Capacity  Report  for  each  site  as 
"Trainees  Unsupportable"  (see  Rueter,  Feldsott,  &  Vaughan,  1989) 

Table  5.5  displays  typical  results  of  the  AFS  328X4  training  capacity  analysis  (for 
representative  site  1).  Note  the  training  deficits  of  18  trainees  imposed  by  Resource  39 
(aircraft),  and  9  trainees  for  Resource  160  (weapons  release  control  system  analyzer),  which 
are  highlighted.  These  results  indicate  that  the  required  amount  of  training  cannot  be 
provided  at  Site  1  because  of  a  lack  of  enough  equipment  needed  to  conduct  training.  A 
similar  analysis  was  conducted  concerning  the  trainer  manpower  needed  to  conduct  required 
AFS  328X4  training  under  the  Current  U&T  scenario.  It  would  appear  that  there  is  little 
or  no  impact  on  the  ability  of  the  field  to  conduct  such  training;  sufficient  personnel  were 
available  for  this  purpose  (or  sufficient  personnel  of  the  next  high  grade  level  were  available 
to  be  substituted  as  OJT  trainers). 

5. 1.2.2  AFS  328X4  Optimization  Analysis  -  Given  the  almost  limitless  number  of 
possible  changes  which  might  be  studied,  a  TDS  analyst  or  functional  user  might  wish  to 
take  another  approach  to  assessing  the  possible  impact  of  specialty  training  changes.  This 
approach  could  make  use  of  the  IOS  optimization  software.  The  analyst  or  user  can  specify 
an  objective  function  or  goal  (such  as  minimization  of  OJT  cost  or  total  training  costs,  or 
maximization  of  the  amount  of  available  equipment,  etc.),  run  the  optimization  program, 
and  examine  the  effects  on  the  specialty  if  the  objective  function  is  maximized  or  minimized. 
The  analyst  can  ask  "What  if'  questions;  for  example: 

What  is  the  impact  on  total  training  costs  if  we  minimize 
initial  resident  course  instruction? 

What  happens  to  specialty  jobs  (proficiency),  if  we  maximize 
FTD  training  and  minimize  OJT?. 

What  is  the  impact  on  proficiency  acquisition  and  training 
costs,  if  we  have  a  10%  cut  in  new  recruits  entering  training? 
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REPRESENTATIVE  SITE:  1 

Training  Capacity:  -  Upper  Bound:  18;  Lower  Bound:  12 


Total  Trainees 

Required:  30 

Resource  Amount 

Amount 

Avail/Req 

Max  Tmees 

Trainees 

Trainees 

ID 

Avail. 

Required 

Ratio 

Supportable 

Required 

Unsupportable 

18 

5840.0 

0.2 

30917.65 

826695. 

26 

0 

27 

1800.0 

7.9 

227.27 

6782. 

29 

0 

39 

208.0 

823.2 

0.25 

_  12, 

2Q 

18  <<<<<<< 

68 

1560.0 

2.0 

5902.30 

111941. 

30 

0 

91 

520.0 

1.9 

277.42 

7543. 

27 

0 

104 

5840.0 

29.8 

195.97 

5864. 

30 

0 

150 

17520.0 

61.6 

284.49 

8658. 

30 

0 

160 

52.0 

77.1 

0.67 

16. 

22 

Jl_«<<<<< 

183 

8760.0 

0.7 

13431.01 

82007. 

6 

0 

Table  5.5  Example  Representative  Site  Training  Capacity  Results:  Current  U&T  Pattern 
(Adapted  from  AFS  328X4  Report,  15  Feb  88,  Table  8.1). 


Some  potential  optimization  problems  become  visible  during  modeling  runs  of  the 
specialty  as  training  constraints  are  identified,  or  possible  new  models  may  be  suggested  by 
initial  optimization  runs.  Other  possible  optimizations  will  be  suggested  by  general  Air  Force 
trends,  such  as  budget  cuts  or  changing  operational  priorities.  In  some  cases,  these  could  be 
complex  problems  with  multiple  values  to  be  optimized. 

The  approach  taken  in  employing  optimization  algorithms  to  solve  maximization  or 
minimization  problems  in  the  TDS  is  to  employ  modular  data  bases  and  seek  solutions  at  the 
lowest  possible  level.  This  isolates  solutions  to  only  the  area  of  interest  and  has  considerable 
efficiency  in  terms  of  saving  computer  time.  Only  the  largest  optimization  problems,  such 
as  minimizing  total  specialty  training  costs,  would  require  employment  of  the  entire  TDS  data 
base.  More  limited  problems  can  thus  be  dealt  with  by  limited  program  runs. 

As  part  of  the  exercise  of  the  TDS  software,  we  tested  the  optimization  software  with  the 
AFS  328X4  data  set.  This  ruction  summarizes  results  of  a  single  TDS  optimization  problem 
as  an  example  of  this  capability. 


81 


The  purpose  of  this  optimization  analysis  was  to  determine  the  numbers  of  hours  on  task 
modules  (TMs)  in  the  FID  course  J4AMF328X4-125,  Carousel  IVE,  which  will  minimize 
total  training  costs,  including  OJT.  This  FTD  course  currently  covers  two  TMs,  73  and  74, 
and  provides  training  on  these  TMs  in  both  the  classroom  and  guided  hands-on  training 
settings.  The  analysis  optimized  hours  for  these  two  TMs  in  these  two  settings.  Table  5.6 
summarizes  the  overall  cost  results  obtained  by  optimization  as  compared  to  the  present 
(Current  U&T  Pattern)  allocation. 


Table  5.6 

FTD  J4AMF328X4-125  Optimization  Results 


Current 

U&T  Pattern 
(baselinel 

Optimization 

Results 

ABR  course  costs 

$  1,676,353 

$  1,676,353 

FTD  -125  costs 

$  73,092 

$  24,097 

All  advanced  course  costs 

$  1,047,943 

$  999,183 

Total  formal  training  costs 

$  2,724,296 

$  2,675,536 

Total  OJT  costs 

$  5,096,500 

$  5,102,513 

Total  OJT  hours 

682,555 

683,398 

As  may  be  seen,  the  overall  costs  associated  with  FTD  -125  were  reduced  by  about 
$49,000.  OJT  costs  went  up  by  about  $6,000,  so  that  the  overall  cost  reduction  was  about 
$43,000. 

Table  5.7  summarizes  the  hours  on  the  selected  TMs,  both  before  and  after  optimization. 


Table  5,7 

Optimization  Results 

_ Setting _ 

Task  Classroom  Hands-on 


Module 

CIM)_ 

Base- 

line 

Optim- 

ization 

Base- 

line 

Optim¬ 

ization 

73 

50  hrs 

1  hr 

98  hrs 

1  hr 

74 

24  hrs 

1  hr 

36  hrs 

46  hrs 
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As  may  be  seen,  the  optimization  analysis  suggests  that  I'M  73  should  be,  in  effect, 
eliminated  from  the  FID  and  that  TM  74  should  be  covered  only  in  the  guided  hands-on 
setting.  The  optimized  FID  -125  had  no  impact  on  OJT  capacity. 

The  baseline,  the  328X4  current  U&T  pattern,  violates  several  OJT  nonlabor  resource 
capacity  constraints.  The  optimized  alternative  U&T  pattern  presented  here  did  not  violate 
any  additional  constraints. 

The  optimized  version  of  FID  course  -125  appears  to  offer  significant  cost  reduction 
possibilities  with  little  impact  on  OJT.  The  nature  of  the  optimization  program  is  such  that 
it  cannot  take  into  account  travel  costs.  The  people  taking  a  course,  and  travel  costs 
associated  with  getting  them  to  the  course,  are  taken  as  fixed  in  the  optimization  run.  In 
this  situation,  the  best  alternative  might  be  to  eliminate  the  course.  This  would  save  the 
travel  costs  associated  with  the  course;  the  travel  costs  saved  might  more  than  make  up  for 
the  additional  OJT  costs. 

This  example  of  Optimization  is  a  very  basic  one,  yet  it  demonstrates  the  capability  of 
the  TDS  to  employ  this  type  of  advanced  technology  to  go  beyond  comparing  predefined 
alternatives  to  actually  formulate  a  new  alternative  U&T  pattern  (based  on  the  outcomes 
of  the  optimization  problem). 

This  type  of  technology  is  most  appropriate  where  the  policy  issue  (or  question  to  be 
answered)  is  an  issue  which  can  be  stated  in  terms  of  some  value  to  be  maximized  or 
minimized. 


5.1.3  AFS  811XX.  Security  and  Law  Enforcement  Specialties 

Basic  data  files  for  the  Security  and  Law  Enforcement  specialties  were  reviewed  and 
some  preliminary  runs  made.  A  few  minor  errors  were  detected  and  corrected;  one  of  these 
errors  involved  the  omission  of  one  entry  point  for  the  Military  Working  Dog  course 
involving  cross  trainees  from  Law  Enforcement  after  their  first  assignment  (course  1025  was 
missing  from  all  811XX  FRMJOB  files).  This  omission  involved  only  one  half  of  one 
percent  of  LE  first  assignment  personnel  so  that  the  impact  of  the  error  is  relatively  minor. 
It  needed  to  be  corrected,  however,  before  further  runs  were  made  for  these  specialties. 

With  three  specialties  and  six  assignment  periods,  the  AFS  811XX  data  files  become  very 
complex  and  difficult  to  quality  control  properly.  Accurate  tracking  of  data  files  and 
changes  under  alternative  models  is  critical  to  successful  data  runs. 

Final  analysis  data  runs  were  completed  for  the  Current  U&T  Pattern  and  six  alternative 
U&T  patterns.  Perhaps  the  most  interesting  results  are  the  comparisons  of  total  training 
costs  for  these  seven  patterns  (using  the  Current  as  a  baseline  to  assess  the  changes),  as 
shown  in  Table  5.8. 
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Table  5.8 


Comparison  of  AFS  811XX  Training  Costs 


Formal 

Course 


U&T 

Training 

OJT 

Total  AFS 

Percent 

Pattern 

Costs  Costs  Training  Costs 

(In  Thousands  of  Dollars) 

Change 

Current  U&T 

24,295 

14,987 

39,282 

Baseline 

Alt.  1* 

31,478 

21,386 

54,864 

+  39.7% 

Alt.  2 

24,814 

15,492 

40,306 

+  2.6% 

Alt.  3 

24,049 

14,947 

38,996 

-  0.7% 

Alt.  4 

22,739 

15,351 

38,990 

-  0.7% 

Alt.  5'* 

24,052 

14,987 

39,039 

-  0.6% 

Alt.  6 

22,493 

13,833 

36,326 

-  7.5% 

Does  not  include  transfer  of  ABGD  training  to  Ft.  Dix,  NJ  (Oct  87) 

**  Calculated  as  a  5%  reduction  of  LE  ABR  attendance  (1%  of  total  formal  school  costs)  with  no 
reduction  in  OJT  costs. 


This  type  of  comparative  display  permits  a  very  concise  evaluation  of  a  set  of  TDS 
analyses  and  provides  a  very  comprehensive  perspective  on  the  impact  of  several  alternative 
proposals.  In  this  case,  the  current  AFS  811XX  U&T  pattern  (as  of  the  time  of  the  TDS 
study  in  1984-85)  involves  a  total  training  cost  of  about  $39  million  per  year,  with  $24 
million  of  that  total  associated  with  formal  training  costs.  Most  of  the  cost  consists  of 
student  and  trainer  manhours. 

Alternative  1,  which  involves  the  greatest  change,  is  a  change  whereby  all  AFS  811XX 
trainees  would  attend  Air  Base  Ground  Defense  (ABGD)  training  prior  to  their  first 
assignment.  This  alternative  was  an  active  proposal  at  the  1985  Training  Planning  Team 
(TPT)  conference  and  was  actually  implemented  as  of  1  October  1987,  when  all  new 
graduates  have  been  sent  to  Ft.  Dix,  NJ,  enroute  to  their  initial  assignments.  The  TDS 
analysis  of  this  alternative  suggests  that  this  policy  change  results  in  a  cost  increase  of  about 
39.7%,  to  almost  $55  million  per  year  (without  considering  the  one-time  costs  of  moving  the 
course  to  Ft.  Dix). 

[Note:  In  a  very  real  sense,  what  was  Alternative  1  has  become  the  new  Current  U&T 
Pattern.  That  is,  it  more  accurately  describes  the  present  situation  for  AFS  811XX 
than  does  the  "Current  U&T  Pattern"  developed  in  the  earlier  TDS  project.  The  other 
alternative  U&T  patterns  would  need  to  be  reformulated  to  include  the  additional 
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training  which  is  now  required  (ABGD  training  and  practice  for  everyone).  For  the 
purposes  of  the  present  project,  such  changes  have  not  been  made  since  the  updating 
of  AFS  files  and  reanalysis  of  alternatives  was  not  a  formal  project  objective.] 

Alternative  2  represents  a  proposal  to  merge  AFSs  811X0  and  811X2  at  the  7-skill  level 
(rather  than  at  the  9-skill  level  at  present).  This  was  modeled  as  a  merged  7-level 
supervisor’s  course  and  free  cross  flow  of  assignments  at  the  sixth  assignment  and  beyond. 
This  change  increases  total  training  costs  by  approximately  $1  million  per  year;  some  of  this 
increase  involves  the  formal  course  (the  new  7-skill  level  course  is  slightly  longer  than  the 
present  separate  courses).  The  remainder  of  the  increase  is  associated  with  increased  OJT 
resulting  for  members  of  either  specialty  cross  flowing  to  assignments  in  the  other  area. 

Alternatives  3-5  have  very  little  impact  on  total  training  costs.  Alternative  3  involves 
splitting  the  Military  Working  Dog  specialty  (presently  AFS  811X2A)  into  two  specialties, 
since  some  jobs  are  more  associated  with  security  flight  operations  (AFS  811X0).  This 
change  would  create  more  complicated  job  flows  but,  as  can  be  seen  in  Table  3.8,  has  very 
little  impact  on  training  costs.  Alternatives  4  and  5  would  transfer  some  administrative  jobs 
to  other  specialties  or  have  those  who  have  Law  Enforcement  training  by-pass  the  ABR; 
these  proposals  change  some  job  and  training  flows  but  have  essentially  no  impact  on 
training  costs. 

Alternative  6  involves  contracting  some  jobs  out,  thus  reducing  the  total  military 
manpower  of  the  specialty.  Some  law  enforcement  and  guard  jobs  are  already  contracted 
at  some  (but  not  all)  Air  Force  Systems  Command  (AFSC)  and  Air  Force  Logistic 
Command  (AFLC)  bases.  Extending  such  contracting  to  all  bases  in  the  noncombatant 
commands  (including  AFLC,  AFSC,  and  Air  Training  Command)  would  save  about  $3 
million  dollars  per  year  in  AFS  811XX  training  costs.  It  is  not  known,  however,  how  much 
training  civilian  guards  on  Air  Force  procedures  would  cost. 

This  type  of  summary  comparison  provides  a  concise  and  understandable  picture  of  the 
impacts  of  the  proposals  on  total  AFS  training  costs.  It  is  a  very  useful  display  which  was 
requested  by  potential  TDS  users  at  the  final  TDS  Progress  Review  (September  1988).  This 
type  of  display  is  created  manually  by  synthesizing  and  condensing  information  from  a  series 
of  TDS  reports. 


5.1.4  AFS  305X4.  Electronic  Computer  and  Switching  Systems  Specialty 

Basic  data  runs  for  the  Electronic  Computer  and  Switching  Systems  specialty  were 
completed  with  no  major  errors  detected.  AFS  305X4  involves  an  extremely  complex  data 
set  since  there  are  multiple  3-skill  level  subspecialties  each  with  its  own  basic  resident 
course  or  FTD  program.  These  subspecialties  merge  at  the  5-skill  level  into  a  common 
specialty,  although  the  subspecialty  jobs  continue  to  exist  through  the  supervisory  level. 
Complete  runs  for  this  specialty  resulted  in  cost  data  comparisons  shown  in  Table  5.9  below: 
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Table  5.9 


Comparison  of  AFS  305X4  Training  Costs  * 


U&T 

Formal 

Course 

Training 

OJT 

Total  AFS 

Percent 

Pattern 

Costs. 

Costs 

Trng  Costs 

Change 

Current 

U&T 

(In  Thousands  of  Dollars) 

2,788  4,926  7,714 

Baseline 

Alt.  1 

2,540 

5,011 

7,551 

-  2.1% 

Alt.  2 

2,900 

5,112 

8,012 

+  3.9% 

Alt.  3 

5,424 

4,065 

9,489 

+  23.0% 

Alt.  4 

3,214 

4,802 

8,016 

+  3.9% 

Alt.  5 

2,569 

5,248 

7,817 

+  13% 

Alt.  6“ 

m 

Data  as  of  July  1988 

’’  Involves  merger  with  other  AFSs  (for  which  there  are  no  TDS  data) 


Note  that  in  the  current  U&T  pattern  for  AFS  305X4  (as  of  1988  when  the  study  was 
finalized)  total  training  costs  over  seven  and  a  half  million  dollars  per  year,  with  the  majority 
of  that  cost  ($4.9  million)  involving  OJT  costs.  AFS  305X4  is  an  extremely  diverse  specialty, 
with  multiple  3-skill  level  shredouts  and  multiple  ABR  courses.  [Note:  subsequent  to  the 
TDS  data  collection  effort,  the  specialty  was  changed  with  Autovon  switching  systems  jobs 
being  moving  to  another  maintenance  specialty,  and  several  shredouts  deleted]. 

Alternative  1  was  a  proposal  to  have  a  common  ABR  course  focusing  on  electronics 
principles  with  most  training  deferred  until  the  first  job.  Under  this  proposal,  formal  course 
training  costs  are  reduced  somewhat  ($248,000)  but  OJT  costs  rise  ordy  slightly  ($85,000),  for 
an  overall  savings  of  2.1%.  This  result  suggests  that  the  present  ABRs  have  little  specific 
equipment  training  and  are  already  focused  on  fundamentals. 

Alternative  2  was  proposed  by  Airborne  Warning  and  Control  Systems  (AWACS)  SMEs 
to  use  only  more  experienced  cross-trainees  in  AWACS  jobs,  thus  avoiding  extensive  AWACS 
training  of  new  recruits.  Such  a  change  would  actually  increase  overall  training  costs  by  some 
$298,000  per  year,  with  that  cost  split  between  formal  courses  and  OJT.  While  this 
alternative  was  preferred  by  AWACS  managers,  it  was  not  popular  among  other  senior 
personnel  of  the  specialty. 
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Alternative  3  was  very  similar  to  Alternative  1  except  that  training  would  be  focused  in 
FTDs  rather  than  OJT.  This  results  in  extensive  cost  growth  in  formal  courses  ($2,636,000) 
with  only  a  moderate  savings  in  OJT  ($861,000).  While  this  is  a  desirable  option  (since  it 
reduces  field  OJT  demands),  it  is  the  most  expensive  alternative  considered. 

Other  alternatives  (4  &  5)  involved  other  training  configurations  and  had  only  moderate 
cost  implications;  SMEs  did  not  consider  either  of  these  alternatives  very  realistic  as  possible 
changes. 

Alternative  6  involved  splitting  the  AFS  with  some  jobs  moving  to  another  AFS  (not 
studied  in  TDS  R&D).  Without  data  from  the  other  AFS,  this  option  could  not  be 
evaluated.  This  restructuring  was  the  option  which  the  Air  Force  actually  decided  to  do; 
the  change  moved  Autovon  Switching  maintenance  jobs  into  another  electronics  specialty 
and  the  remaining  jobs  were  redesignated. 


52  Troubleshoot,  Enhance  and  Refine  TDS  Software 


As  noted  earlier,  TDS  software  and  data  files  were  reviewed  to  detect  errors  or  anomalies 
which  might  suggest  software  problems.  In  addition,  new  versions  of  several  TDS  programs 
were  developed,  as  well  as  several  new  programs  which  are  ready  for  implementation  on  the 
AFHRL  UNISYS.  We  have  also  defined  some  new  and  updated  data  files.  The  purpose 
of  this  section  is  to  summarize  those  updates. 

5.2.1  Updated  and  New  Programs: 

UTPSIM  -  Two  situations  were  identified  in  which  errors  in  input  data  files  could  cause 
erratic  program  behavior.  In  both  cases,  the  impact  on  program  output  was  very  small,  so 
the  program  was  not  rerun.  The  program  has  been  updated  to  detect  these  circumstances. 
In  one  case,  the  program  can  make  a  correction.  In  the  other  case,  the  program  prints  an 
error  message.  This  latter  case  arises  when  an  entry  (e.g.,  job  or  training  course)  is  missing 
from  the  FRMJOB  file  (LN12-nnn09a.).  The  problem  was  that  course  25  (1025)  was 
missing  from  all  811XX  FRMJOB  files.  This  problem  should  be  corrected  in  all  such  files 
on  the  UNISYS  (e.g.,  LN12-81109.,  LN12-811093.).  UTPSIM  has  been  updated  on  the 
UNISYS. 

TRNPRF  -  This  program  has  been  updated  to  correct  a  small  error  which  caused  OJT 
counts  to  be  incremented  occasionally  when  no  OJT  hours  were  being  given  and  to  output 
a  new  data  file  which  is  used  in  the  linear  programming  capacity  analysis.  The  impact  of 
the  error  on  program  output  was  small;  previous  runs  do  not  need  to  be  rerun.  The  new 
data  file,  described  below,  contains  OJT  hours  given  on  each  TM  to  each  simulated  airman 
in  each  job;  this  file  may  be  thought  of  as  an  extension  to  the  entity  history  file  1 .  will  be 
called  the  ENTOJT  file  or  entity  OJT  hours  file.  The  TRNPRF  has  been  updated  on  the 
UNISYS;  however,  there  is  still  a  need  to  update  the  ECL  generation  facility  for  the  new 
data  file. 


87 


RCSCRO  -  This  program  estimates  hours  of  nonlabor  resources  required  for  OJT.  It 
corrects  a  small  problem  with  the  version  of  this  program  on  the  VAX  version  of  the 
program.  A  complete  new  version  of  this  program  has  been  loaded  to  the  UNISYS. 

RCSCRAT  -  This  program  does  ratio  capacity  analysis  for  nonlabor  resources  in  OJT. 
It  has  been  updated  to  improve  the  output  report  formats  and  to  provide  a  new  data  file 
needed  for  linear  programming  capacity  analysis.  Three  data  files  produced  by  the  old 
version  of  this  program  are  obsolete.  A  new  version  of  the  program  has  been  loaded  to  the 
UNISYS. 

RCSCRCT  -  In  general,  training  resource  quantities  are  estimated  by  applying  linear 
regression  equations  to  training  hours  output  from  TRNPRF.  Such  an  equation  is  needed 
for  each  TM  by  resource  (student  labor  hour,  instructor  preparation  labor  hours,  instructor 
delivery  labor  hours,  non  labor  resources)  by  setting  combination.  For  some  such 
combinations,  no  equation  is  available  (for  lack  of  data  estimates).  The  old  versions  of  all 
RCS  resource  requirement  estimation  programs  estimated  zero  resource  hours  when  no 
regression  equation  was  available.  In  the  case  of  student  labor  hours,  it  makes  more  sense 
to  use  the  TRNPRF  training  hours  than  a  zero  value  for  TM-by-setting  combinations  with 
no  regression  equations.  The  new  version  of  RCSCRCT  implements  this  approach  for 
student  labor  hours  in  non-OJT  settings.  A  new  version  of  the  program  is  now  on  the 
UNISYS. 

RCSCRQCT  -  This  program  implements  the  improvement  described  above  for  RCSCRCT 
for  student  OJT  labor  hours.  A  new  version  of  the  program  has  been  loaded  to  the 
UNISYS. 

RCSCOSCS  -  This  is  a  new  program  which  estimates  cost  per  student  week  of  training 
(in  contrast  to  the  existing  cost  program,  which  estimates  annual  costs).  A  new  version  of 
the  program  has  been  loaded  to  the  UNISYS. 

RCSMCCOS  -  This  is  a  new  program  which  estimates  annual  OJT  costs  for  the  major 
commands.  This  program  allocates  OJT  costs  (currently  built  up  by  job  and  representative 
site)  to  the  major  commands.  It  requires  a  new  data  file  for  each  AFS  which  contains 
percentages  of  airmen  at  each  representative  site  that  are  in  each  major  command.  These 
files  for  the  four  AFSs,  named  MAJPCT.nnn,  have  been  created  and  loaded  to  the 
UNISYS. 

RCSCTRAQ  -  This  program  applies  the  ratio  capacity  analysis  method  for  instructors  in 
OJT.  It  takes  into  account  instructor  grade  level,  based  on  average  grade  level  of  trainee, 
and  instructor  substitutability  (e.g.,  higher-grade  instructors  can  be  used  in  place  or  required 
lower  grades,  but  the  reverse  is  probably  not  true).  A  new  version  of  the  program  has  been 
loaded  to  the  UNISYS. 

UNCAP  -  This  program  applies  the  linear  programming  capacity  analysis  method  for 
nonlabor  resources  in  OJT.  The  program  requires  three  new  input  files.  These  are:  (a)  a 
file  output  from  the  ratio  capacity  program  (RCSCRAT)  which  specified  which 
representative  sites  and  resources  are  to  be  analyzed,  (b)  a  file  which  specified  which  TMs 
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are  to  be  included,  and  (c)  a  file,  output  from  TRNPRF,  which  contains  data  on  OJT  hours 
by  TM  for  each  simulated  airman.  In  addition,  the  RCS  run  parameter  file 
(LN12-nnnR01a)  contains  some  additional  parameters.  The  source  code  for  this  program 
has  been  loaded  to  the  UNISYS:  LINCAP  (main  program);  CLOX,  CLOXOOOl, 
CLOX0002,  and  ZSXPHD  (sub-  routines  called  by  LINCAP),  and  UNPACK!  (subroutines 
called  by  ZSXPHD).  This  program  will  not  be  fully  implemented  on  the  UNISYS  until  new 
program  documentation  has  been  written.  It  also  requires  the  updated  versions  of  TRNPRF 
and  RCSCRAT. 

5.2.2  Updated  and  New  Data  Files: 

Entity  OJT  Hours  File  -  This  new  file  is  output  from  the  updated  version  of  TRNPRF. 
It  contains  OJT  hours  by  TM  for  each  simulated  airman  and  job  in  the  simulation.  This  file 
is  used  for  linear  programming  capacity  analysis  and  may  eventually  be  used  by  all  RCS 
programs. 

MAJCOM  Percentage  File  -  This  file  contains  MAJCOM  percentages  for  representative 
sites  and  is  used  to  estimate  MAJCOM  OJT  costs. 

Ratio  Capacity  Short  Resources  File  -  This  file  identifies  resources  that  are  short  at 
representative  sites.  It  is  output  from  the  nonlabor  resource  OJT  ratio  capacity  analysis 
program  and  input  to  the  linear  programming  capacity  analysis  program. 

Ratio  .Capacity  Short  Resource  XM  Eik  -  This  file  lists  TMs  which  require  specified 
resources.  It  is  output  from  the  LPCTTM  file,  based  on  an  input  ratio  capacity  short 
resource  file.  This  file  is  input  to  the  linear  programming  capacity  analysis  program. 

RCS  Run  Parameter  File  (LN12-nnnR01a!  -  This  file  contains  several  additional 
parameters  that  are  used  by  the  linear  programming  capacity  analysis  program.  The  file 
need  not  be  updated  for  use  by  other  RCS  programs  except  the  major  command  costing 
program. 

5.2.3  New  Procedure  for  Estimating  FTP  Travel  Costs 

In  the  original  version  of  the  TDS,  each  FID  course  was  assumed  to  be  held  at  the 
primary  base  designated  to  provide  such  training  in  AFR  50*5.  Travel  costs  were  estimated 
using  the  assumption  that  personnel  in  other  units  traveled  to  the  primary  FID  location  for 
training.  To  enhance  and  refine  the  operation  of  the  TDS,  the  capability  to  estimate  travel 
costs  when  an  FTD  course  is  held  at  more  than  one  base  (as  is  the  case  for  many  FID 
courses)  has  been  developed.  The  procedure  operates  as  follows. 

For  each  base  where  airmen  who  need  to  take  a  particular  FTD  course  are  stationed,  the 
travel  cost  between  that  base  and  each  of  the  bases  where  the  FTD  course  is  being  offered 
(hereafter  referred  to  as  the  FTD  base  is  computed.  Die  FTD  base  with  the  lowest 
computed  travel  cost  is  then  selected  and  the  simulated  airmen  are  assigned  to  that  FTD 
location.  When  the  procedure  is  completed  for  all  bases  with  airmen  who  need  to  take  the 
FTD  course,  the  FID  bases  are  ordered  according  to  the  number  of  airmen  assigned  to 
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them.  If  the  FTD  base  with  the  fewest  number  of  airmen  assigned  to  take  the  FTD  has  at 
least  the  minimum  number  of  airmen  required  to  hold  a  course,  then  the  procedure  is 
complete.  The  "minimum  number"  can  be  specified  by  the  analyst  (Currently,  the  number 
"2"  is  used  as  a  default  value).  However,  if  this  criterion  is  not  satisfied,  the  FTD  base  with 
the  fewest  number  of  airmen  assigned  to  it  is  eliminated  (i.e.,  the  FTD  course  is  assumed  not 
to  be  taught  there)  and  those  airmen  are  reassigned  to  the  next  least  expensive  FTD  base. 
This  procedure  continues  until  all  FTD  bases  have  at  least  the  minimum  number  of  airmen 
required  to  hold  a  course. 

53  Discussion  of  Constrained  TDY-TO-SCHOOL  Resources 

In  earlier  sections  of  this  report,  some  preliminary  discussion  of  the  possible  reduction 
of  25  percent  in  TDY-to-school  funds  was  included  within  the  analysis  of  AFSs  423X1  and 
328X4.  As  noted  in  those  sections,  the  initial  analyses  were  incomplete;  they  lacked  a  review 
of  the  impact  on  unit  OJT  capacity  (pending  having  revised  capacity  software  operational). 
Until  the  OJT  capacity  analyses  revisions  were  operational,  any  extended  discussion  of  the 
impact  of  the  proposed  reduction  in  TDY  funds  was  not  feasible.  A  short  review  of  the 
capacity  calculations  is  warranted. 

The  Training  Capacity  Component  of  the  RCS  systematically  compares  the  estimated 
resource  availabilities  for  the  representative  sites  of  an  AFS  to  the  total  amounts  of  resources 
required  at  those  sites  to  accommodate  the  specific  volumes  of  training  in  particular  training 
states  that  are  designated  for  those  sites  within  the  U&T  patterns  developed  in  the  FUS. 
This  comparison  first  involves  computing  the  ratio  between  the  total  amount  of  each  resource 
that  is  available  for  the  provision  of  training  at  each  site  and  the  total  amount  of  the  resource 
that  is  actually  needed  to  provide  all  of  the  training  designated  for  the  site  within  a  particular 
U&T  pattern.  For  provision  of  the  designated  amount  of  training  to  be  unequivocally 
feasible,  the  calculated  ratios  for  all  required  resources  must  uniformly  be  greater  than  or 
equal  to  1.0. 

Conversely,  if  a  ratio  less  than  1.0  is  calculated  for  any  required  resource,  it  is  infeasible 
to  conduct  all  designated  training  at  the  site.  Moreover,  if  ratios  less  than  1.0  are  calculated 
for  two  or  more  required  resources,  the  cumulative  effect  of  the  concurrent  shortages  of 
those  resources  on  the  capacity  of  the  site  to  provide  the  required  training  will,  in  general, 
depend  upon  the  feasibility  of  concentrating  those  resource  shortage  effects  on  a  restricted 
group  of  trainees. 

Mathematical  programming  provides  a  means  for  evaluating  the  feasibility  of  allocating 
training  resources  in  such  a  way  that  the  trainees  to  whom  training  is  not  provided  are  so 
concentrated.  This  involves  calculating  the  number  of  people  who  should  be  trained  in  each 
training  state  at  each  site  so  that  the  total  number  of  people  who  receive  their  designated 
training  is  maximized  and,  at  the  same  time,  certain  constraining  conditions  are  satisfied. 
The  applicable  constraints  include,  most  notably,  that  no  training  in  excess  of  the  amount 
designated  in  the  U&T  pattern  is  conducted  in  any  training  state,  and  that  no  more  than  the 
total  amount  of  each  resource  available  for  the  provision  of  training  is  used  at  any  site. 
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The  results  obtained  by  applying  this  method  of  training  capacity  estimation  to  evaluate 
the  training  capacity  of  representative  site  1  for  AFS  328X4  for  both  the  current  U&T 
pattern  and  the  reduced  TDY-to-School  funding  scenario  are  summarized  in  Table  5.10. 
Results  are  presented  jointly  to  facilitate  a  comparative  analysis  of  these  two  U&T  patterns 
to  show  how  training  constraints  are  identified  and  quantified.  These  results  were  derived 
within  the  context  of  a  linear  programming  formulation  of  the  estimation  method,  as 
discussed  above. 

Section  A  indicates  the  total  number  of  trainees,  3.41  per  year  on  average,  for  whom  the 
provision  of  training  at  the  site  has  been  designated  by  the  FUS.  Next,  Section  B  compares 
the  estimated  amounts  of  individual  resources  available  at  the  site  and  the  estimated 
amounts  of  those  resources  required  for  the  provision  of  all  training  designated  for  the  site 
by  the  FUS.  The  first  and  second  columns  of  Section  B  indicate  the  specific  U&T  patterns 
and  resources  to  which  the  estimates  relate.  The  third  and  fourth  columns  contain  the 
estimates  derived  in  the  RCS  for  the  amounts  of  those  resources  available  and  required,  for 
providing  the  training  required  for  the  representative  site  within  the  U&T  pattern.  The  fifth 
column  for  each  U&T  pattern  presents  the  ratios  of  the  amounts  available  to  the  amounts 
required  for  individual  resources.  A  ratio  greater  than  or  equal  to  1.0  indicates  that  the 
amount  of  the  resource  available  at  the  site  is  sufficient  to  provide  the  total  amount  of 
training  needed  to  sustain  the  U&T  pattern.  Conversely,  a  ratio  less  than  1.0  means  that 
there  is  a  shortage  of  the  resource  at  the  site. 

Section  C.  reports  the  estimate  of  training  capacity  derived  for  each  U&T  pattern  by 
means  of  linear  programming.  These  estimates  indicate  the  numbers  of  trainees,  2.89  and 
2.88  per  year  on  average,  to  whom  all  designated  training  for  the  two  U&T  patterns  can  be 
provided  with  the  amounts  of  resources  available  at  the  site.  Thus,  in  combination,  the 
estimates  presented  in  Sections  A  and  C  state  that  for  units  characterized  by  this 
representative  site,  0.52  and  0.53  trainees  per  year,  on  average,  will  not  receive  some  portion 
of  the  training  designated  for  them  within  these  two  U&T  pattern  scenarios.  Thus,  we  can 
see  that  the  Reduced  TDY-to-  School  change  has  had  very  little  (essentially  no)  impact  on 
OJT  capacity. 

The  linear  programming  analysis  also  identifies  the  specific  resource  that  imposes  the 
most  restrictive  limitation  on  the  capacity  of  the  representative  site  to  accomplish  all  of  the 
training  designated  within  the  U&T  pattern.  Accordingly,  for  the  two  resources  for  which 
the  total  amounts  available  are  insufficient  for  providing  all  of  the  training  designated  for 
the  representative  site,  the  estimates  reported  in  Section  D  of  Table  5.10  indicate  that  for 
both  U&T  patterns,  the  total  amount  of  resource  39  (aircraft)  that  is  available  for  providing 
training  is  equal  to  the  total  amount  required  for  providing  the  maximum  achievable 
quantity  of  training,  whereas  the  total  amount  available  of  resource  160  (weapons  release 
control  system  analyzer)  exceeds  the  amount  required  for  providing  the  maximum  achievable 
quantity  of  training  by  17.1  units  for  the  current  U&T  and  12.1  units  for  the  reduced 
TDY-to-School  U&T  alternative.  Thus,  in  this  situation,  resource  39  imposes  the  most 
restrictive  constraint  on  the  training  capacity  of  the  representative  site. 
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Table  S.10 


Estimated  OJT  Capacities  for  'typical  Units 
Characterized  by  Representative  Site  1 

A.  Total  trainees  requiring  training:  3.41 

B.  Resource  Constraints  limiting  die  provision  of  required  training: 

Ratio  of 


U&T  Pattern 

Resource 

Amount 

Available 

Amount 

Required 

Availability 
to  Requirements 

Current  U&T 

39 

208.0 

823.2 

0.25 

160 

52.0 

77.1 

0.67 

188 

1506.0 

77.0 

20.27 

Reduced  TDY-to- 

School  Funding 

39 

208.0 

829.8 

0.25 

160 

52.0 

80.8 

0.64 

188 

1560/0 

77.7 

20.09 

C.  Linear  Programming  Estimates  of  Training  Capacity: 

Current  U&T  Pattern  2.89 

Reduced  TDY-to-School  U&T  Pattern2.88 


D.  Resource  Constraints  Limiting  Maximum  Amount  of  Training  Achievable: 

U&TJEattom  Resource  Amount  Available  Amount  Required 


Current  U&T 

39 

208.0 

208.0 

160 

52.0 

34.9 

Reduced  TDY 

39 

208.0 

208.0 

160 

52.0 

39.9 

This  type  of  information,  considered  site  by  site,  must  be  evaluated  along  with  the  formal 
training  and  OJT  costs  in  any  comparison  of  two  or  more  alternative  U&T  models.  All  of 
these  data  are  available  for  managers  use  in  assessing  the  possible  consequences  of  changing 
AFS  training  programs,  jobs,  and  utilization  policies. 
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5.4  Summaiy  and  Recommendations 


An  extensive  exercising  of  the  TDS  software  has  been  completed  for  the  four  TDS  R&D 
specialties  including  OJT  capacity  analyses.  Results  of  these  exercises  indicate  the  system 
is  operating  well  and  produces  detailed  and  very  useful  reports  and  data. 

In  addition  to  the  work  originally  planned,  training  cost  reports  for  new  policy  issues 
involving  the  two  maintenance  specialties  were  given  priority,  at  the  request  of 
AFHRL/EDET,  to  assure  the  availability  of  information  for  briefing  the  MATAG  in 
mid-May  1989.  Results  for  a  reduced  TDY-to-School  U&T  pattern  as  well  as  the  current 
U&T  for  the  two  maintenance  specialties  were  carefully  checked  and  appear  reasonable. 
A  new  U&T  pattern  involving  a  25%  reduction  in  ABR  flow  was  developed  to  meet  an 
additional  AFHRL/IDET  request  and  cost  reports  for  the  two  specialties  completed.  These 
data  were  delivered  to  AFHRL  on  12  May  1989,  and  were  used  in  briefing  the  MATAG. 
This  exercise  demonstrated  a  relatively  quick  response  capability  for  answering  queries  from 
functional  managers  concerning  AFSs  studied  in  the  TDS  R&D,  and  thus  was  an  extremely 
valuable  addition  to  the  work  planned  for  the  present  project. 

TDS  software  troubleshooting  resulted  in  a  number  of  minor  changes  to  FUS  and  RCS 
programs;  most  changes  have  little  if  any  impact  on  AFS  data  files  or  reports.  Some 
programs  were  revised  to  improve  efficiency  or  correct  minor  problems;  some  additional 
programs  were  written  to  provide  internal  quality  control  checks  and  greater  flexibility. 

RCS  software  has  been  substantially  upgraded  and  most  of  the  revised  programs  are  now 
available  on  the  UNISYS  at  AFHRL,  although  not  all  have  yet  been  completely  tested  and 
debugged.  These  major  upgrades  include  improvements  in  the  OJT  training  capacity 
program,  new  capability  to  summarize  data  by  MAJCOM,  and  a  substantial  revision  of  FIT) 
travel  costing  procedures.  The  latter  program  is  operational  and  has  been  tested  on  the 
CONSAD  VAX  but  has  not  yet  been  tested  on  the  UNISYS;  it  must  also  undergo  further 
quality  control  evaluation  by  MDC  and  Metrica  analysts  and  programmers.  We  recommend 
such  testing  and  quality  control  be  undertaken  in  the  near  fature. 

Optimization  software  was  tested  in  a  limited  optimization  problem  on  one  FTD  of  AFS 
328X4  training  programs.  This  test  indicated  the  system  is  now  operational  (on  the  MDC 
VAX)  and  demonstrated  the  approach  needed  to  utilize  this  system  capability.  Additional 
tests  are  needed  before  the  optimization  software  is  made  fully  operational  on  the  UNISYS. 
We  recommend  such  testing  be  include  in  any  future  TDS  work. 

Overall,  the  TDS  Software  Exercise  was  a  very  successful  undertaking.  Some  problems 
with  data  files  and  programs  were  discovered  and  corrected.  In  addition,  such  problems  led 
to  the  design  and  implementation  of  system  improvements.  However,  due  to  additional 
system  exercises  (Reduced  TDY-to-School  and  reduced  ABR  scenarios)  and  other  activities 
(TDS  training  for  AFHRL  analysts),  some  improvements  in  product  formats  and  subsystem 
programs  could  not  be  completed  within  the  framework  of  the  present  project  We 
recommend  that  these  improvements  be  included  in  any  future  TDS  task  work  or  within  the 
context  of  future  TDS  R&D  projects. 
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6.0  SENSITIVITY  ANALYSIS 


The  TDS  Integration  and  Optimization  Subsystem  (IOS)  modeling  soft  ware  was 
subjected  to  an  extensive  series  of  sensitivity  analyses.  These  sensitivity  analyses  examined 
the  effects  of  seven  input  variable  classes  on  three  output  variables.  The  analyses  were  done 
so  that  main  effects  of  the  seven  input  variable  classes  could  be  assessed,  as  well  and  two- 
and  three-way  interactions  of  the  input  variable  classes.  This  sensitivity  analysis  extends  the 
sensitivity  analysis  conducted  earlier  under  the  TDS  proof-of-concept  contract.  The  earlier 
work  involved  sensitivity  analyses  for  the  328X4,  Radar  and  Inertial  Navigation  Systems 
specialty.  The  present  work  involved  similar  analyses  with  the  423X1,  Aircraft  Environmental 
Systems  specialty,  the  305X4,  Computer  and  Electronic  Switching  Systems  specialty,  and  the 
811XX,  Security  Police  specialties. 

The  purpose  of  these  sensitivity  analyses  was  to  investigate  the  effects  of  variations  in 
input  parameters  on  the  operation  of  program  objective  functions.  More  specifically,  the 
sensitivity  analyses  had  three  objectives:  a)  to  identify  and  rank  input  variables  which  affect 
objective  program  functions,  b)  to  identify  and  rank  any  two-  and  three-way  interactions 
between  and  among  input  variables  which  affect  objective  program  functions  with  the  main 
effects  of  singular  variables,  and  c)  to  fully  evaluate  the  effects  of  the  entire  range  of  the 
input  variables  on  each  objective  function. 


6.1  Methods 


In  these  sensitivity  analyses,  seven  classes  of  input  variable  were  manipulated,  including 
both  Field  Utilization  Subsystem  (FUS)  utilization  and  training  (U&T)  pattern  modeling 
inputs  and  Resource/Cost  Subsystem  (RCS)  resource  requirements  and  cost  modeling  inputs. 
Current  U&T  pattern  values  were  used  as  a  baseline.  Inputs  were  systematically  increased 
and  decreased  for  each  input  variable  class.  Effects  of  these  changes  and  interactions  among 
these  changes  were  determined  on  total  formal  training  costs,  total  on-the-job  (OJT)  training 
costs,  and  total  training  costs. 


The  overall  TDS  U&T  pattern  and  cost  modeling  functions  are  embodied  in  the 
following  computer  programs: 


1.  UTPSIM: 

2.  TRNPRF: 

3.  RCSCRCT: 

4.  RCSCROCT: 

5.  RCSCRCI: 

6.  RCSCROCI: 

7.  RCSCOSTS: 


U&T  pattern  simulation 
OJT  estimation 

Student  formal  training  labor  hour  resource  requirement  estimation 
Student  OJT  labor  hour  resource  requirement  estimation 
Instructor  formal  training  labor  hour  resource  requirement  estimation 
Instructor  OJT  labor  hour  resource  requirement  estimation 
Formal  training  and  OJT  cost  estimation 
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Several  other  TDS  programs  are  concerning  with  training  capacity  estimation, 
optimization,  and  utility  functions.  The  sensitivity  analyses  reported  here  focused  on  training 
cost  estimation  and  on  the  programs  listed  above.  These  programs  and  their  various  inputs 
and  outputs  are  documented  in  the  TDS  User  Manual  and  the  TDS  Programmers  Guide 
developed  for  this  project  (see  Preface). 

Seven  input  variable  classes  were  manipulated.  Three  sets  of  values  were  used  for  each 
class:  current  U&T  pattern  values  and  two  other  sets.  These  input  variable  classes  and  their 
values  are  described  below: 

Random  number  seed  -  The  U&T  pattern  simulation  (UTPSIM),"  which  estimates  job  and 
training  flows,  is  a  monte  carlo  simulation.  It  uses  a  stream  of  pseudorandom  numbers 
generated  by  the  computer  determine  the  direction  taken  by  each  simulated  entity  at  each 
choice  point.  The  random  number  stream  used  for  a  computer  run  is  determined  by  a  seed 
which  is  input.  Three  different  random  number  streams  were  used  for  the  328X4  and  423X1 
specialties,  determined  by  three  seed  values.  For  the  305X4  and  811XX  specialties,  two 
random  number  streams  were  used,  determined  by  two  seed  values. 

Job  transition  probabilities  -  A  set  of  transition  probabilities  input  to  the  UTPSIM 
program  are  used  to  determine  the  job,  between-job  training,  or  leave  state  that  simulated 
airmen  go  to  from  each  job  or  between-job  training.  The  set  of  transition  probabilities  from 
a  particular  job  must  sum  to  100%.  In  addition  to  current  U&T  pattern  values,  a  set  of 
values  was  created  by  increasing  large  probabilities  (greater  than  20%)  by  10%  (e.g., 
multiplying  them  by  1.1)  and  decreasing  small  probabilities  by  10%  (e.g.,  multiplying  them 
by  .9).  Similarly,  a  third  set  of  transition  probabilities  was  created  from  the  current  U&T 
pattern  values  by  decreasing  large  probabilities  by  10%  and  increasing  small  values  by  10%. 

Job-driven  training  transition  probabilities  -  Job-driven  training  transition  probabilities 
are  used  by  the  UTPSIM  program  to  assign  simulated  airmen  to  training  courses  taken  while 
in  jobs  (e.g.,  field  training  detachment  courses).  Since  an  airman  can  possibly  take  several 
job-driven  training  courses  while  in  particular  job,  these  probabilities  need  not  sum  to  exactly 
100%.  Modified  sets  of  these  probabilities  were  obtained  by  increasing  all  current  U&T 
pattern  values  by  10%  (e.g.,  multiplying  by  1.1)  and  by  decreasing  all  current  U&T  pattern 
values  by  10  %  (e.g.,  multiplying  by  .9).  In  U&T  pattern  models  for  the  811XX  specialties, 
most  training  courses  were  treated  as  job  and  time  driven  rather  than  job  driven.  Thus,  for 
the  811XX  specialty,  job  and  time  driven  training  transition  probabilities  were  modified 
instead  of  job-driven  training  transition  probabilities,  using  the  procedure  described  here. 

Allocation  curve  parameters  -  Allocation  curves,  in  the  TDS  model,  define  the 
relationships  between  training  hours  and  proficiency  for  each  task  module  (TM)  and  training 
setting.  These  curves  are  modeled  by  second  order  polynomial  equations;  each  TM-setting 
combination  has  two  parameters  (the  additive  constant  is  always  zero).  The  allocation  curve 
parameters  are  used  by  the  TRNPRF  program  to  estimation  OJT  hours  needed  to  achieve 
full  proficiency.  Modified  sets  of  these  parameters  were  obtained  by  increasing  all  current 
U&T  pattern  values  by  10%  and  by  decreasing  all  current  U&T  pattern  values  by  10%. 
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Training  TM  hours  -  For  each  formal  training  course,  hours  devoted  to  each  TM  in  each 
training  setting  are  input  to  the  TRNPRF  program.  These  data  are  used  to  account  for 
proficiency  obtained  by  formal  training  in  determining  OJT  quantities.  Modified  sets  of  these 
parameters  were  obtained  by  increasing  all  current  U&T  pattern  values  by  10%  and  by 
decreasing  all  current  U&T  pattern  values  by  10%. 

Student  resource  requirement  model  parameters  -  For  RCS  purposes,  student  labor  hours 
(instruction  and  preparation)  required  for  training  programs  are  estimated  (programs 
RCSCRCT  and  RCSCROCT).  This  is  done  using  a  two-parameter  linear  model  for  each 
TM  and  training  setting.  Ses  if  modified  parameters  were  obtained  by  increasing  all  current 
U&T  pattern  values  by  10%  and  by  decreasing  all  current  U&T  pattern  values  by  10%. 

Instructor  resource  requirement  model  parameters  -  For  RCS  purposes,  instructor  labor 
hours  (instruction,  preparation,  and  administration)  required  for  training  programs  are 
estimated  (programs  RCSCRCI  and  RCSCROCI).  This  is  done  using  a  two-parameter  linear 
model  for  each  TM  and  training  setting.  Modified  sets  of  these  parameters  were  obtained 
by  increasing  all  current  U&T  pattern  values  by  10%  and  by  decreasing  all  current  U&T 
pattern  values  by  10%. 

These  seven  input  variable  classes  with  three  sets  of  input  values  for  each  class  define 
a  3X3X3X3X3X3X3  factorial  experimental  design  (2X3X3X3X3X3X3  for  305X4  and 
811XX).  One  complete  replication  of  this  design  would  require  2187  TDS  modeling  runs. 
Since  it  was  not  necessary  to  assess  interactions  beyond  the  second  order,  it  was  not 
necessary  to  make  all  of  these  runs.  In  fact,  1256  modeling  runs  were  made  for  328X4, 1215 
runs  were  made  for  423X1,  and  810  runs  were  made  for  305X4  (fewer  runs  were  needed  for 
305X4  and  811XX  because  only  two  levels  were  studied  for  the  random  number  stream). 
These  runs  permitted  all  interactions  through  the  second  order  to  be  assessed,  with  some 
residual  to  evaluate  the  overall  impact  of  higher  order  interactions.  For  each  selected  cell 
in  the  design,  a  complete  TDS  cost  modeling  run  was  made.  The  total  formal  training  cost, 
total  OJT  cost,  and  total  training  cost  (sum  of  formal  training  cost  and  OJT  cost)  was 
retained  for  analysis. 


6 2  Results 


6.2.1  AFS  328X4 

For  completeness  and  comparative  purposes,  results  for  the  328X4  specialty  are 
summarized  here.  Table  6.1  presents  proportions  of  variance  attributable  all  main  effects 
and  to  those  higher-order  effects  which  accounted  for  at  least  1%  of  the  total  variance. 

Detailed  ANOVA  results  for  AFS  328X4  are  presented  in  CDRL  CLIN0001:33  from  the 
proof-of-concept  TDS  contract.  Table  6.2  presents  within-group  means  on  each  outcome 
variable  for  levels  associated  with  all  main  effects  and  with  the  job  transition  probability  by 
job-driven  training  transition  probability  interaction  effect. 
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The  AFS  328X4  results  contained  in  Tables  6.1  and  62  were  summarized  from  the 
previous  sensitivity  analysis  report  Some  additional  analyses  were  done  for  the  present 
project.  These  results  provide  a  better  basis  for  evaluating  the  actual  magnitudes  of  the 
impacts  on  training  costs  associated  with  the  independent  variables  in  the  sensitivity  analysis. 
Consider,  first,  the  random  number  stream  or  seed  factor.  In  the  context  of  the  sensitivity 
study  design,  this  is  a  random  effect. 

The  actual  seed  values  used  were  sampled  from  a  population  of  possible  values;  The 
training  cost  effect  magnitude  of  this  factor  can  be  measured  by  the  cost  variance  associated 
with  the  effect  As  shown  in  Table  6.1,  the  random  number  seed  effect  accounted  for  very 
little  of  the  overall  cost  variance.  Unbiased  estimates  of  the  training  cost  standard  deviation 
associated  with  the  random  number  seed  factor,  for  formal  training,  4320;  for  OJT,  36523, 
and  for  total  costs,  40452.  Under  the  usual  assumptions,  one  would  expect  costs  to  vary  by 
as  much  as  plus  or  minus  these  amounts  about  68%  of  the  time  due  to  changes  in  the 
random  number  stream  used. 


Total  Formal 

Total  OJT 

Total 

Training  Cnst 

-Cast _ 

TarihgCbst 

Effect 

Ra 

E* 

R* 

Random  number  seed  (A) 

Job  transition  probabilities  (B) 

.02 

.02 

Training  transition 

probabilities  (C) 

.10 

.02 

.05 

Allocation  curve 

parameters  (D) 

35 

30 

Training  TM  hours  (E) 

38 

.02 

.02 

Student  resource  requirement 

parameters  (F) 

38 

.45 

36 

Instructor  resource  require 

parameters  (G) 

.04 

.03 

BC 

.10 

.06 

.10 

Note:  When  no  R2  value  is  given,  the  value  is  less  than  .01 

Table  6.1  AFS  328X4  ANOVA  Summary 
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-10% 


Normal 


±10% 


Random  number  seed  (A) 


Formal  training 

2744450 

2754270 

2747990 

OJT 

5095300 

5130830 

5055100 

Total  training 

2744450 

7891100 

7803090 

Job  transition  probabilities  (B) 

Formal  training 

2745190 

2700720 

2732230 

OJT 

5104310 

5133000 

4999520 

Total  training 

790951 

7900390 

7731700 

Job-driven  training 
transition  probabilities  (C) 

Formal  training 

2030000 

2742220 

2800270 

OJT 

4974710 

5130200 

5124470 

Total  training 

7004720 

7878480 

7984740 

Allocation  carve  parameters  (D) 

Formal  training 

2748920 

2750050 

2748450 

OJT 

5428100 

5110230 

4753010 

Total  training 

8170770 

7800290 

7501470 

Training  TM  hours  (E) 

Formal  training 

2503140 

2750000 

2931970 

OJT 

5180030 

5110050 

5002700 

Total  training 

7743170 

7800000 

7934740 

Student  resource  require¬ 
ment  parameters  (F) 

Formal  training 

2552120 

2758210 

2930940 

OJT 

4090180 

5110230 

5481080 

Total  training 

7248300 

7808840 

8418030 

Instructor  resource  re¬ 
quirement  parameters  (G) 

Formal  training 

2724990 

2759430 

2703780 

OJT 

4974050 

5109810 

5204440 

Total  training 

7099240 

7809250 

7908230 

B  by  C  (columns  reflect  levels  of  C) 

B1  (-10%): 

Formal  training 

2342730 

2802820 

2111100 

OJT 

4048270 

5324500 

5290700 

Total  training 

0991010 

8127380 

8201920 

B2  (Normal) 

Formal  training 

2744910 

2744890 

2849220 

OJT 

5137700 

5143880 

5103320 

Total  training 

7882070 

7888780 

7952540 

B3  (+10%) 

Formal  training 

2700300 

2705880 

2808340 

OJT 

5000000 

5024020 

4941288 

Total  training 

7712370 

7729910 

774900 

Table  62  AFS  328X4  WIthin-level  Training  Cost  Means 

The  remaining  independent  variables  in  the  sensitivity  analysis  are  fixed  effects.  For 
these  effects,  the  regression  parameter  associated  with  the  linear  contrast  for  an  effect 
estimates  the  amount  of  cost  change  associated  with  a  10%  change  in  the  factor  value  (since 
levels  of  the  factors  were  produced  by  changing  the  actual  values  by  10%,  as  described 
above).  These  regression  parameters  are  presented  in  Table  63. 
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Formal 

Training 

On-the-job 

Training 

Total 

Training 

Job  transition  probabilities  (B) 

-  3717 

-  79068 

-  82780 

Job-driven  training 
transition  probabilities  (C) 

113454 

69966 

183420 

Allocation  curve  parameters  (D) 

238 

-338168 

-338405 

Training  TM  hours  (E) 

183659 

-  90019 

93640 

Student  resource  requirement 
parameters  (F) 

192072 

393199 

585270 

Instructor  resource  parameters  (G) 

19599 

114897 

134496 

Table  6.3  AFS  328X4  Main  Effect  Linear  Regression  Parameters 


AF£  423X1  Remits 

Table  6.4  summarizes  the  proportions  of  variance  accounted  for  by  all  main  effects  and 
other  effects  which  account  for  at  least  1%  of  the  cost 


Total  Formal 

Total  OJT 

Total 

Training  Cost 

Cost 

Training  Cost 

Effect 

B2 

R* 

Random  number  seed  (A) 

Job  transition  probabilities  (B) 

.01 

.01 

.02 

Training  transition  probabilities  (C) 

38 

.01 

.02 

Allocation  curve  parameters  (D) 

.03 

.03 

Training  TM  hours  (E) 

24 

.08 

Student  resource  parameters  (F) 

29 

33 

.64 

Instructor  resource  parameters  (G) 

.02 

.03 

.03 

AB 

.02 

.03 

.03 

Note:  When  no  R*  value  is  given,  the  value  is  less  than  .01 

Table  6.4  AFS  423X1  ANOVA  Summaiy 


Table  65  presents  training  cost  within-cell  means  for  all  main  effects  and  for  the  random 
number  seed  by  job  transition  probability  interaction.  For  all  three  training  costs,  unbiased 
estimates  of  variance  associated  with  the  random  number  seed  factor  were  negative. 
Maximum  likelihood  estimates  of  these  variances  were  zero.  While  these  variances  are 
probably  not  in  fact  zero,  they  are  very  low.  Table  6.6  presents  regression  parameters 
associated  with  linear  contrasts  for  the  fixed  effects  in  the  sensitivity  analysis. 


99 


-it* 

Normal 

41*% 

— 

■amber  mod  (A) 

Koran! 

tnrialan 

2*8*237 

21*9533 

2*97944 

OUT 

54727*3 

54934*4 

54*58*8 

Total  tn 

75*2*2* 

74*3194 

75*3752 

Jo*  tra 

probaMi 

tin  (B) 

Foraal  i 

InUag 

212*279 

2*9*272 

2*78143 

OJT 

55M2S4 

5493221 

542475* 

Total  tn 

7*345*3 

7591493 

75*2913 

JMrin 

mSSLmc) 

Foraal ! 

tnMag 

190*11 

2*9*451 

221*251 

CUT 

551334* 

54834*8 

54355** 

Total  tn 

75*1351 

75*1*59 

7*45759 

*»(D) 

Foraal! 

Tahrfag 

2*9*94* 

2*9*72* 

3mm 

CUT 

5534315 

554*87* 

538579* 

Total  tn 

7*332(4 

7*4559* 

74*4745 

TkaUacTM  boon  (E) 

Foraal  I 

2*21537 

219*72 * 

217434* 

OJT 

55*01* 

554*87* 

5151293 

Total  tn 

759*35* 

7*45597 

7527*54 

OUT 

Total  tn 


(F) 


2M2333 

5146145 

7MM77 


547747* 

75MM1 


msm 


Table  6.5  AFS  423X1  Within-level  Training  Cost  Means 
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Formal 

Training 

On-the-job 

IramJsg 

Total 

Training 

Job  transition  probabilities  (B) 

-  21058 

-44767 

-65825 

Job-driven  training 
transition  probabilities  (C) 

111120 

-38916 

72204 

Allocation  curve 
parameters  (D) 

0 

-74260 

-  74260 

Training  TM  hours  (E) 

77412 

-108763 

-31351 

Student  resource  requirement 
parameters  (F) 

96529 

331248 

427777 

Instructor  resource  requirement 
parameters  (G) 

11967 

216544 

228514 

Table  6.6  AFS  423X1  Main  Effect  Linear  Regression  Parameters 

AFS  305X4  Results 

Table  6.7  summarizes  the  proportions  of  variance  accounted  for  by  all  main  effects  and 
other  effects  which  accounted  for  at  least  1%  of  the  cost  variance.  Table  6.8  presents 
training  cost  within-cell  means  for  all  main  effects. 


Total  Formal 
Training  Cost 

Total  OJT 
Cost 

Total 

Training  Cost 

R2 

Effect 

R2 

R2 

Job  transition  probabilities  (B) 

.03 

32 

.16 

Training  transition  probabilities  (C) 

.13 

.03 

Allocation  curve  parameters  (D) 

.15 

.06 

Training  TM  hours  (E) 

.50 

.11 

Student  resource  requirement 
parameters  (F) 

39 

.43 

51 

Instructor  resource  require¬ 
ment  parameters  (G) 

.12 

.07 

Note:  When  no  R2  value  is  given,  the  value  is  less  than  .01 


Table  6.7  AFS  305X4  ANOVA  Summary 
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Unbiased  estimates  of  the  training  cost  standard  deviation  associated  with  the  random 
number  seed  factor,  for  formal  training,  8662;  for  OJT,  6678,  and  for  total  costs,  the 
estimate  was  negative.  Under  the  usual  assumptions,  one  would  expect  costs  to  vary  by  as 
much  as  plus  or  minus  these  amounts  about  68%  of  the  time  due  to  changes  in  the  random 
number  stream  used. 


-10% 

Normal 

±1Q% 

Random  number  seed  (A) 

Formal  training 

3081153 

3093470 

OJT 

4920877 

4811326 

Total  training 

7902031 

7904603 

Job  transition  probabilities  (B) 

Formal  training 

3158842 

3078248 

3024845 

OJT 

5020032 

4833949 

4594034 

Total  training 

8178874 

7912197 

7618879 

Job-driven  training 
transition  probabilities  (C) 

Formal  training 

2950852 

3087440 

3223644 

OJT 

4836446 

4815929 

4795640 

Total  training 

7787299 

7903369 

8019284 

Allocation  curve  parameters  (D) 

Formal  training 

3087328 

3087247 

3087328 

OJT 

46450(7 

4876915 

4956488 

Total  training 

7732395 

7964161 

8043817 

Training  TM  hours  (E) 

Formal  training 

2850498 

3087247 

3324159 

OJT 

4828844 

4876915 

4772711 

Total  training 

7679342 

7964162 

8096869 

Student  resource 
requirement  parameters  (F) 

Formal  training 

2880975 

3087371 

3293590 

OJT 

4513049 

4816016 

5118950 

Total  training 

7679343 

7964162 

8412540 

Instructor  resource 
requirement  parameters  (G) 

Formal  training 

3064247 

3087314 

3110375 

OJT 

4654502 

4816150 

4977363 

Total  training 

7718749 

7903464 

8087738 

Table  6.8  305X4  Within-level  Training  Cost  Means 
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Table  6.9  presents  regression  parameters  associated  with  linear  contrasts  for  the  fixed 
effects  in  the  sensitivity  analysis. 


Formal 

Training 

On-the-job 

Training 

Total 

Training 

Job  transition  probabilities  (B) 

-66999 

-212999 

-279997 

Job-driven  training  transition 
probabilities  (C) 

136396 

-20403 

115992 

Allocation  curve  parameters  (D) 

0 

155711 

155711 

Training  TM  hours  (E) 

236830 

-  28067 

208763 

Student  resource  requirement 
parameters  (F) 

206308 

302950 

509258 

Instructor  resource  requirement 
parameters  (G) 

23064 

161430 

184494 

Table  6.9  AFS  305X4  Main  Effect  Linear  Regression  Parameters 


6.3  Summary  and  Recommendations 

In  general,  the  results  for  AFS  423X1  and  305X4  specialties  replicate  the  sensitivity 
analysis  findings  for  AFS  328X4.  First,  random  number  streams  have  very  little  impact  on 
the  bottom  line  training  cost  estimates. 

In  only  one  case  did  an  effect  involving  the  random  number  stream  factor  accounted  for 
as  much  as  1%  of  the  cost  variance.  IDS  results  are  generally  stable  over  different  random 
number  streams.  Large  differences  between  TDS  analyses  for  different  U&T  patterns  can 
be  attributed  to  real  differences  in  the  U&T  patterns,  rather  than  simply  to  random  variation 
due  to  different  random  number  sequences.  This  is  a  very  important  finding~that  TDS 
model  results  are  not  sensitive  to  particular  random  number  sets  used. 

Another  nonsignificant  result  of  note  is  that  for  interactions.  Of  the  168  two-  and 
three-way  interactions  for  the  three  specialties,  only  two  accounted  for  greater  than  1%  of 
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training  cost  variance.  The  residual  beyond  the  main  effects  and  two-  and  three-way 
interactions  was  also  less  than  1%,  showing  that  no  higher-order  interactions  are  important. 
Studying  high-order  interactions  in  sensitivity  analyses  such  as  the  present  one  requires  much 
effort.  This  is  because  many  model  runs  are  needed.  However,  if  only  main  effects  and 
low-order  interactions  are  to  be  studied,  many  fewer  model  runs  are  needed.  This  permits 
such  sensitivity  analyses  to  be  done  with  much  less  effort  and  permits  many  more  input 
variables  to  be  studied.  It  also  permits  simple  estimation  of  impacts  on  costs  for  other 
changes  in  the  independent  variables  studied  here.  We  recommend  that  future  TDS 
sensitivity  analyses  focus  on  main  effects  and  perhaps  on  two-way  interactions,  rather  than 
on  higher-order  interactions. 

For  formal  training,  the  same  variables  were  most  important  for  all  three  specialties. 
These  key  variables  are  the  training  TM  hours  and  the  student  resource  requirement  model 
parameters.  Our  previous  experience  with  economic  analysis  of  formal  training  suggests  that 
student  labor  hours  are  the  major  cost  contributor  to  classroom  training.  In  the  TDS  model, 
student  labor  hours  are  function  of  the  training  TM  hours  as  modified  by  the  student 
resource  requirement  model. 

For  OJT,  the  results  were  less  consistent  across  the  specialties.  In  all  specialties,  the 
most  important  factor  were  the  student  resource  requirement  model  parameters.  In  AFS 
328X4,  file  allocation  curve  parameters  were  also  important.  In  AFS  423X1,  the  training  TM 
hours  were  important  for  OJT  costs  as  well  as  for  formal  training  costs.  In  AFS  305X4  the 
job  transition  probabilities  were  important,  while  the  allocation  curve  parameters  and  the 
instructor  resource  requirement  model  parameters  were  of  smaller,  but  practically  significant, 
importance. 

For  total  training  costs,  the  results  were  moderately  consistent  across  specialties;  student 
resource  requirement  model  parameters  were  the  dominant  factor.  For  two  of  the 
specialties,  the  training  TM  hours  were  important.  For  the  third  specialty,  allocation  curve 
parameters  were  important.  Overall,  it  is  clear  that  the  TDS  cost  model  is  very  sensitive  to 
the  student  resource  requirement  model  parameters.  Given  these  results,  we  recommend 
that  emphasis  in  future  work  be  placed  on  testing  improved  methods  for  estimating  these 
parameters. 

The  present  sensitivity  analysis  looked  at  seven  of  the  most  important  TDS  input  variable 
classes.  The  TDS  model  has  many  other  input  variables.  We  recommend  that  these 
additional  variables  be  examined  in  future  sensitivity  analyses.  These  future  studies  can  be 
streamlined  by  focusing  only  on  main  effects. 
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7.  TRAINING  DECISIONS  TECHNOLOGY  POTENTIAL  APPLICATIONS 

The  Training  Decisions  technologies  developed  during  the  TDS  R&D  project  have  a 
number  of  potential  applications  which  meet  and  go  beyond  the  original  IDS  design 
parameters.  Many  of  these  are  detailed  in  the  TDS  Fined  Report  (see  Vaughan,  et  al.,  1989), 
and  other  TDS  documentation  (see  TDS  User  Guide ,  etc.).  Some  of  the  advanced  potential 
applications  were  outlined  in  earlier  chapters,  as  they  relate  to  current  AFHRL  R&D 
projects.  Additional  possible  applications  not  in  the  present  AFHRL  research  program  are 
noted  below. 

7.1  Current  and  Future  Research  on  Training  Management  Issues 

By  its  very  nature,  the  TDS  involves  the  quantification  of  training  requirements,  and 
any  research  or  development  efforts  in  the  area  of  improving  Training  Management  or 
decision  making  are  potential  applications  for  the  TDS  technology.  A  number  of  studies  of 
Air  Force  training  have  indicated  major  problems  and  need  for  improvements.  In  describing 
the  current  training  management  system,  Johnson  and  his  coworkers  note  that  there  are 
considerable  constraints  on  improvements  of  Air  Force  training  management  These  include 
"system  fractionalization,"  unpredictable  production  goals,  "vague  assessment  measures,"  and 
limited  investments  in  training  resources  (Johnson,  Green,  Soldwisch,  Turner,  and  Wall, 
1989:51-52). 

One  of  the  joint  ATC-HSD  projects  to  improve  Air  Force  Training  Management  is 
the  Advanced  Training  System  (ATS),  a  computer-based  management  system  which  will 
automate  most  training  management  functions  and  facilitate  the  flow  of  information  among 
ATC  organizations.  The  present  ATS  design,  however,  does  not  include  sufficient  advanced 
training  decisions  support  technology  to  solve  some  of  the  present  major  problems. 

HSD  commissioned  a  study  to  identify  areas  of  potential  improvement  in  Air  Force 
training  management.  Some  of  the  priority  needs  identified  in  the  HSD  study  include: 

1.  An  algorithm  to  constrain  and  objectively  prioritize  training 
requirements  in  the  MAJCOMs  could  both  reduce  the  personnel 
energy  expended  and  enhance  the  requirement  screening 
responsiveness. 

2.  Models  and  data  flows  to  improve  Manpower,  Personnel,  and 
Training  (MPT)  issue  leverage,  accuracy,  and  timeliness  in 
acquisition  planning  would  directly  impact  mission  readiness  with 
new  systems. 

3.  Systems  for  accurate  officer  and  pilot  selection  and 
classification  would  reduce  recruiting  efforts  and  pipeline 
through-put 
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4.  Development  of  job  specific  task  baselines  for  military,  OJT, 
and  aircrew  instruction  would  ensure  war-fighting  competencies 
and  dampen  much  of  the  inefficient  perturbations  in  instructional 
goals. 

5.  Objective  measures  for  performance  assessment  for  rated  and 
enlisted  personnel  would  enable  managers  to  demonstrate  impacts 
from  changes  in  training  system  operations  and  increase  training 
effectiveness. 

6.  A  system  to  quantify,  capture,  and  articulate  total  resource 
costs  for  training  would  support  a  resource  conservative 
stabilization  in  the  training  system  across  the  USAF. 

7.  Consolidation  and  automation  of  OJT  record  keeping  would 
decrease  supervisor  diversion  of  effort  and  increase  qualify  time 
with  apprentices. 

(Johnson,  Green,  Soldwisch,  Turner,  and  Wall,  1989:51-52) 


In  a  second  volume  of  the  HSD  study,  Johnson  and  his  co-workers  recommend  an 
integrated  strategy  to  identify  technological  "shortfalls"  needed  to  remedy  major  training 
management  problems  in  the  "21st  century  environment"  They  observe: 


An  examination  of  current  and  planned  USAF  research  activity 
that  is  relevant  to  training  management  against  the  system  needs 
reveals  several  shortfalls  in  reaching  desired  system  capabilities. 
Most  notable  needs  are  the  US  Air  Force-wide  general  data  base 
network,  a  single  task  list  development  system,  a  dynamic  job 
survey  system,  hierarchical  task  structure  system,  an  integrated 
physiological-psychological  student  selection  system,  and  a  total 
USAF  costing  system. 

(Johnson,  Damewood,  and  Turner,  1989:52) 


Clearly,  several  of  these  areas  are  ones  where  the  TDS  R&D  has  already  made  some 
significant  progress.  Comprehensive  reports  of  the  TDS  R&D  were  not  available  to  Johnson 
and  his  coworkers  at  the  time  of  their  study.  Several  of  the  areas  they  suggest  are  also  ones 
where  advanced  TDS  work  could  be  beneficial  As  noted  in  the  last  chapter,  however,  such 
advanced  research  needs  to  be  very  carefully  integrated  so  that  R&D  outcomes  can  be  easily 
operationalized  and  research  results  can  be  made  synergistic. 
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The  TDS  could  be  a  key  part  of  this  advanced  training  decisions  support  system;  it  has 
the  necessary  capabilities  for  AFS  modeling,  assessment  of  planned  changes,  and  evaluation 
of  total  AFS  training  costs  and  OJT  capacities.  No  other  system  yet  has  such  capabilities. 
Linked  with  AOTS  and  the  performance  measurement  methodologies  developed  by  AFHRL, 
TDS  could  provide  part  of  the  foundation  needed  to  considerably  enhance  Air  Force 
managers’  capabilities  for  efficient,  well-integrated  MPT  decision  making. 


12  New  Weapon  Systems  Acquisition  Process 

One  area  which  deserves  special  comment  is  that  of  estimating  new  MPT  requirements 
for  new  weapon  systems  under  development.  TDS  could  be  particularly  useful  in  determining 
and  costing  alternative  ways  to  meet  training  requirements  for  such  systems.  This  adaptation 
for  Training  Decisions  Technology  will  require  additional  development  effort,  but  would 
bring  the  considerable  advances  made  in  the  TDS  R&D  to  bear  on  this  problem  area. 

Currently,  McDonnell  Douglas  has  an  Internal  Research  and  Development  (IRAD) 
effort  underway  in  the  potential  application  of  TDS  technology  for  the  new  weapons  system 
acquisition  process.  Results  to  date  suggest  that  some  new  techniques  will  be  required  for 
the  prediction  of  allocation  curves  from  known  or  estimated  task  (task  module) 
characteristics,  since  no  SMEs  with  experience  in  training  the  new  weapon  system  would  be 
available  as  a  data  source.  This  and  other  predictive  issues  are  presently  being  studied,  and 
one  effort  is  currently  underway  to  determine  whether  realistic  predictions  of  Task  Module 
allocation  curves  can  be  achieved. 

While  significant  technological  development  issues  are  involved,  work  to  date  suggests 
that  TDS  technology  can  be  meaningfully  applied  in  the  new  weapons  systems  acquisition 
area.  Thus,  this  is  an  area  where  significant  additional  R&D  investment  appears  worthwhile. 


13  Training  Decisions  Technology  for  Other  DOD  Agencies 

Another  major  potential  application  for  Training  Decisions  Technologies  may  be  in  the 
other  military  services  and  related  DOD  agencies.  Since  most  of  the  other  services  and 
agencies  have  task-based  job  analysis  systems,  the  foundation  for  the  application  of  TDS 
technology  exists.  In  addition,  the  present  interservice  training  programs  may  provide  the 
vehicle  for  adapting  and  extending  TDS  technology  to  examine  interservice  issues.  The 
present  TDS  focuses  attention  on  just  Air  Force-funded  training;  however,  given  access  to 
appropriate  MPT  and  cost  information  from  the  other  services,  it  should  be  possible  to  utilize 
IDS  technology  to  estimate  interservice  training  costs  and  to  model  job  and  training  flow 
patterns  as  well. 

The  basis  for  such  extension  may  already  exist  in  the  form  of  interservice  occupational 
analysis  projects  conducted  jointly  by  the  Air  Force,  Army,  and  Navy.  Interservice  and  joint 
occupational  projects  have  been  completed  for  a  number  of  areas  (computer  operators  and 
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programmers,  explosive  ordinance  disposal,  cooks  and  bakers,  etc.);  such  joint  efforts  have 
created  a  situation  where  compatible  data  collection  formats  and  joint  analysis  procedures 
are  already  developed.  Thus,  this  cooperative  network  could  be  used  for  administration  of 
new  task  characteristic  data  and  U&T  pattern  information,  rather  than  having  to  establish 
an  entirely  new  system. 

Some  new  technologies  would  need  to  be  developed  to  insure  the  compatibility  of 
occupational  data  and  to  reconcile  differences  in  data  collection  techniques  and  procedures. 
Such  new  technological  development  would  be  minimal,  however,  because  of  the  close 
compatibility  of  the  existing  interservice  programs. 


7.4  Training  Decisions  Technology  in  the  Civilian  Sector 

Business  and  Industry  are  now  becoming  conscious  of  the  tremendous  dollar  investment 
they  have  been  making  in  on-the-job  training  programs,  which  heretofore  have  largely  been 
hidden  costs.  The  Training  Decisions  Modeling  technologies  could  be  applied  quite  cost 
effectively  wherever  a  large  corporation  or  other  organization  has  a  well  developed 
task-based  job  analysis  data  base,  preferably  one  where  CODAP  or  ASCII  CODAP 
processing  are  available. 

In  a  broader  context,  the  type  of  occupational  modeling  developed  for  IDS  could  be 
applied  fairly  efficiently  wherever  sufficient  job  analysis  and  training  data  are  presently 
available.  There  is  sufficient  generality  to  this  approach  to  decision  making  that  it  could  be 
realistically  applied  almost  anywhere  that  appropriate  data  could  be  gathered  (any  large 
industry,  union,  educational  institution,  etc.).  Furthermore,  the  TDS  as  a  decision  support 
system,  is  a  very  positive  model  for  other  human  resources  management  decision  making. 


108 


8.0  CONCLUSIONS  AND  RECOMMENDATIONS 


This  examination  of  the  technologies  involved  in  the  proof-of-concept  Training 
Decisions  System  research  and  development  suggests  that  it  is  a  well-developed,  viable 
decision  support  system  (DSS)  with  significant  potential  applications  not  only  for  training 
decision  making  but  also  to  the  broader  arena  of  MPT  modeling  and  policy  decisions. 
However,  based  on  the  sensitivity  analyses  and  exercises  of  the  system,  there  remain  a 
number  of  areas  within  the  present  TDS  design  where  additional  development  and 
refinement  are  needed  to  fully  exploit  its  potential.  Some  recent  findings  from  other  R&D 
efforts  might  also  be  integrated  to  form  the  basis  for  an  integrated  MPT  modeling  capability. 
In  addition,  further  exploratory  R&D  would  be  worthwhile  to  help  us  fully  understand  the 
significant  analytic  and  decision  support  potentials  of  this  system. 


8.1  TDS  as  an  Integrated  Training  Decision  Technology 

The  TDS  proof-of-concept  system  appears  to  have  successfully  integrated  several  rather 
disparate  technologies  (operations  research  modeling,  occupational  analysis  task  clustering 
and  specialty  analysis,  cost  accounting  and  prediction,  etc.)  into  a  coherent  and  serviceable 
training  DSS.  The  independent  technologies  are  in  and  of  themselves  significant 
contributions  to  the  state-of-the-art  in  their  respective  areas;  however,  their  integration  into 
a  single  operating  system  provides  training  decision  makers  with  a  more  comprehensive  data 
analysis  capability  than  has  previously  been  available.  For  the  first  time,  we  can  develop  a 
realistic  estimate  of  the  total  annual  costs  of  training  (including  OJT)  in  a  specialty  and 
assess  the  capacity  of  representative  units  (organizations)  to  conduct  training. 

One  of  the  most  significant  lessons  learned  from  this  R&D  project  was  the  need  for  a 
comprehensive  "model"  of  each  occupation  (Air  Force  specialty)  to  insure  a  common 
understanding  among  decision  makers  as  to  the  jobs  and  training  programs  of  the  AFS.  Such 
a  model  must  be  comprehensive  in  its  coverage  of  jobs  and  training  programs,  yet  simple 
enough  to  be  easily  understood  and  manipulated.  It  must  present  a  dynamic  picture  of  the 
occupation,  one  which  represents  the  flows  of  individuals  through  jobs  and  training  states 
over  the  full  span  of  their  careers. 

In  the  proof-of-concept  TDS  design,  this  need  for  comprehensive  yet  dynamic  modeling 
was  successfully  met,  yet  its  ability  to  portray  the  intricacies  of  the  more  complex  specialties 
varied  depending  to  die  degree  of  complexity  of  the  AES  model.  We  do  not  yet  know  the 
"optimum"  level  of  detail  for  representing  the  more  complex  specialties  (in  terms  of  the 
number  of  Task  Modules  or  the  number  of  jobs  needed  to  capture  the  essential  elements  of 
the  AFS).  Further  R&D  is  recommended  to  vary  these  aspects  of  TDS  models  in  order  to 
determine  the  most  appropriate  level  of  analysis  for  critical  decision  making. 

In  addition,  further  refinement  of  data  gathering  is  required.  Various  data  collection 
efforts  in  the  proof-of-concept  TDS  R&D  project  varied  in  the  degree  of  success  in  capturing 
needed  data.  In  particular,  some  of  the  most  innovative  data  collection  efforts,  such  as  the 
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Resource  Requirements  Survey  and  Resource  Availability  Survey,  were  constrained  in  terms 
of  time  and  development  effort  possible  at  that  stage  of  the  project.  Yet  these  data  elements 
proved  to  be  among  the  most  critical  elements  in  predicting  total  training  costs  and  the 
capacities  of  representative  units  to  conduct  on-the-job  training.  Thus,  we  recommend  that 
these  data  collection  instruments  and  procedures  be  targeted  for  further  improvement. 


82  TDS  &  STATE-OF-THE-ART  SCIENCE 

A  review  of  the  technologies  involved  in  the  TDS  in  terms  of  their  relationship  to 
state-of-the-art  developments  of  several  scientific  disciplines  indicated  that  some  TDS 
technologies  are  extremely  innovative.  Indeed,  several  of  these  areas  (such  as  dynamic 
occupational  modeling)  have  potential  applications  which  go  well  beyond  there  use  in  support 
of  training  decision  making.  Perhaps  the  most  innovative  aspect  of  the  TDS  proof-of-concept 
R&D  was  its  integration  of  several  independent  technologies  into  a  single  decision  support 
system.  The  success  of  this  type  of  integration  suggests  that  other  state-of-the-art 
technologies  might  also  be  added  to  future  TDS  improvements  to  take  advantage  of  new 
techniques  and  data  bases.  The  modular  design  of  the  proof-of-concept  TDS  will  greatly 
facilitate  such  integration  of  new  developments. 

In  some  cases,  the  TDS  requirement  for  new  technology  and  procedures  has  resulted 
in  the  development  and  refinement  of  other  systems  which  go  well  beyond  the  original  TDS 
requirement.  One  example  of  this  is  the  task  clustering  technology  developed  in  the  AFHRL 
refinement  of  ASCII  CODAP,  where  an  advanced  capability  now  exists  for  the  identification 
and  interpretation  of  inherent  task  clusters  (modifies).  Since  this  new  technology  was 
developed  after  the  point  in  time  where  it  might  have  been  used  in  the  TDS  R&D  project, 
such  new  techniques  have  not  yet  been  fully  tested  as  to  their  utility  for  supporting  new  TDS 
studies.  We  recommend  that  any  future  TDS  R&D  include  provision  for  testing  the 
usefulness  of  the  new  ASCII  CODAP  technologies  for  identifying  and  interpreting  task 
modules.  In  addition,  other  recently-developed  technologies  need  to  be  monitored  for 
potential  applications  in  future  TDS  development 


8.3  POTENTIAL  INTERFACE  WITH  OTHER  R&D  PROJECTS 

The  possible  interaction  and  integration  of  the  TDS  technology  with  other  AFHRL 
R&D  projects  were  examined  in  some  detail.  The  conclusion  drawn  from  this  examination 
was  that  several  of  these  R&D  efforts  have  the  potential  of  enhancing  TDS  operations  and, 
conversely,  TDS  has  potential  value  for  those  projects.  Some  type  of  systematic  interface 
needs  to  be  developed  where  procedures  and  data  derived  in  one  AFHRL  project  can  find 
immediate  application  in  other  R&D  efforts.  TDS  might  be  used  or  adapted  for  a  variety 
of  purposes  involving  the  modeling  of  Air  Force  specialties.  There  is  also  a  need  to  examine 
the  potential  of  TDS  for  use  as  a  basis  for  integrating  a  number  of  these  MPT  research 
initiatives  into  a  comprehensive  MPT  modeling  system. 
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8.4  TDS  EXERCISES 


One  of  the  major  efforts  under  the  present  tasking  was  to  exercise  the  system  and  its 
software  to  insure  proper  operation  and  to  identify  needed  improvements  in  software  and 
procedures.  The  system  exercise  was  a  very  successful  undertaking  and  a  number  of 
problems  were  identified  and  corrected.  However,  not  all  of  the  improvements  in  product 
formats  and  subsystem  programs  could  be  completed  within  the  framework  of  the  present 
project.  We  recommend  these  improvements  be  included  in  any  future  TDS  R&D  project 


8.5  TDS  DOCUMENTATION 

Another  major  effort  under  the  present  tasking  was  the  redevelopment  and  refinement 
of  TDS  documentation.  The  TDS  Procedural  Guide:  TDS  User  Instructions,  August  1988,  was 
separated  into  two  volumes:  the  TDS  User  Manual  and  the  TDS  Programmer  Guide,  both 
dated  1  November  1989.  In  addition,  to  satisfy  a  need  for  concrete  examples  to  support  both 
of  these  procedural  guides,  a  third  and  fourth  volume  of  TDS  Examples  were  developed 
which  include  examples  of  data  collection  instruments  ( Examples  -  Volume  1),  TDS  data  files, 
runstreams,  products  and  reports  ( Examples  -  Volume  2)  for  both  a  Current  U&T  Pattern 
and  one  Alternative  U&T  pattern.  This  compilation  of  materials  represents  an  extremely 
complex  and  comprehensive  portrayal  of  what  the  TDS  is  and  how  it  operates. 

These  revised  TDS  documents  should  prove  to  be  useful  in  communicating  the  structure 
and  operation  of  the  TDS  to  potential  users  and  other  interested  parties.  These  volumes 
have  already  been  used  by  staff  members  of  Search  Technology,  Inc.,  of  Norcross,  GA,  as  a 
basis  for  developing  a  computer-based  TDS  tutorial  (demonstrator)  system.  This  tutorial  is 
aimed  at  providing  potential  TDS  users  with  an  automated  way  to  learn  about  the  TDS  at 
their  own  pace,  with  assistance  provided  (via  menu  selections)  for  additional  explanations  or 
discussions  as  needed.  Under  an  extension  to  the  present  task  order,  members  of  the  TDS 
R&D  team  provided  assistance  to  the  Search  Technology  staff  in  reviewing  their  TDS  data 
displays,  providing  definitions  and  examples  of  analysis  outcomes,  and  offering  suggestions 
for  improving  the  tutorial  displays. 


8.6  SENSITIVITY  ANALYSES 

In  general,  the  results  of  sensitivity  analyses  conducted  under  the  present  tasking  were 
highly  satisfactory  (as  discussed  earlier  in  Chapter  6).  The  system  was  found  to  be  relatively 
insensitive  to  variability  in  random  number  streams  and  to  two-  and  three-way  interactions 
among  variables.  This  provides  some  validation  for  the  approach  taken  to  the  simulation  of 
specialty  job  and  training  flows  and  the  quantification  of  APS  training  requirements.  It  also 
suggests  that  further  sensitivity  analyses  could  focus  on  main  effects  and  thus  more  input 
variables  could  be  assessed  economically.  The  present  sensitivity  analyses  looked  at  seven 
of  the  most  important  TDS  input  variable  classes.  The  TDS  model  has  many  other  input 
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variables  which  may  be  important.  We  recommend  that  future  TDS  sensitivity  analyses  focus 
on  main  effects  and  that  additional  variables  be  studied. 


8.7  ADDITIONAL  TDS  R&D 

Based  on  the  results  of  this  review  of  TDS  technologies,  the  sensitivity  analyses,  and 
exercising  of  the  system,  several  additional  R&D  efforts  appear  worthwhile.  These  include 
additional  work  to  validate  allocation  data  (and  curves),  as  well  as  some  assessment  of 
potential  data  bias.  One  tasking  in  the  present  project  was  to  complete  preliminary  research 
plans  for  both  an  allocation  data  validation  study  and  a  data  bias  assessment.  These  research 
plans  provide  details  for  these  suggested  studies  as  a  next  step  toward  improving  the  TDS 
data  collection  procedures  and  operational  data  bases.  We  recommend  that  these  two 
studies  be  included  in  any  future  training  decisions  R&D  at  the  earliest  possible  opportunity. 
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